Thursday, February 21, 2008

Foundations of 21st century journalism

Up to this point, I've been talking about some of the concepts that shape my thinking about media and journalism: Quality; epistemology; the cultural flaws that warp these discussions; the ideas that I think will lead us around those obstacles to answers. But I haven't really offered any answers of my own.

One reason: What use are they? They're not scholarly. They're not researched and footnoted. By the standards that I respect, my answers are little more than jackleg bullshit.

So to put the proper value on these answers, understand that each comes with endless caveats and a deep sense of humility. The future isn't known, everything can (and will) change in unexpected ways, and the odds of me being right on more than half of these statements is absurdly low. I accept that.

But if you'll take these ideas in that spirit -- not as confident proclamations, but as working insights and imagination gained through hard experience, reading and conversation -- then perhaps you'll find their intrinsic value. If they turn out to be predictively true -- great. If they wind up having no value of their own, but spark your thinking toward something great, then I'm OK with that, too.

Human intelligence is based on prediction. We shouldn't be afraid to predict, just as we shouldn't perpetually tie our egos to being correct.

Here goes:

MONOCULTURE TO ECOSYSTEM
Modern media is in transition from the monoculture monopolies of the 20th century to the diverse explosion of expression that represents the 21st century future. Think of the past as a wood-pulp tree farm; think of the future as a rainforest. To outsiders, a tree farm appears orderly and logical while a rainforest appears impassible, chaotic and dangerous. To residents, however, a rainforest is a vibrant, living ecosystem, and a tree farm is artificial and sterile.

We are moving into a media ecosystem of multiple niches and processes. No single niche dominates, and despite constant competition within and between niches, the resulting chaos is externally stable because of its healthy inner dynamism. All the ideas expressed here are to be conceived as existing within the new media ecosystem. To those who fear the thought of a news media that is not controlled by trained elites, here's your lesson for the day: Control doesn't scale.

STUCTURED AND SEMI-STRUCTURED DATA
The No. 1 functional shift, from a future media historian's perspective, will be the change from the current "document" mindset to the future "database" mindset. Modern journalists see themselves as people who report facts and write stories and cannot grasp that the act of turning information into narrative instantly limits its usefulness and accuracy. News writers will continue to create narratives, because the human brain sorts and stores data by narrative mnemonics. In fact, the bulk of the information we consume will continue to be structured into narrative.

But the primary function of newsgathering organizations will be to create and curate semi-structured databases of interesting/significant information. This, by the way, is the reason that the expansion of The Semantic Web matters right now.

SCALABILITY
Narrative doesn't scale to a global information economy. Personal insight about a candidate doesn't scale to a national campaign. A fair and balanced examination of a two-sided story doesn't scale to a topic like global climate disruption. News media will shift from an artificial one-size-fits-all system (based on front pages, production schedules, newshole, broadcast formats, standard server/bandwidth configuration, etc.) to one that expands and contracts depending on situations. This will require new tools and conventions. Many of these solutions will be social.

OPEN SOURCE
Proprietary, compiled information tools and repositories will fail to keep pace with their open source competitors. Ultimately, all first-class news platforms will be based on open-source principles, and all commonly held information will be structurally compatible.

INFORMATICS
Informatics is the study of the structure of information. "Discovery Informatics" uses sophisticated software agents to detect and explore patterns in enormous streams and vast pools of data. Since all major news organizations will have comparable news databases, much 21st century newsmedia competition will consist of duels over user-tools. The news company with the "best" informatics tools stands a good chance of being the commercial winner.

THE BLUR: NEWS, INFORMATION AND ADVERTISING
Artificial distinctions between information types will be blurred and then forgotten. The new challenge will be getting the right information to the right user at the proper time, rather than maintaining firewalls or winnowing out things that "aren't news anymore." In the future, it's all about the end-user's needs and experience. This means that something other than artificial firewalls will have to stand as a credibility marker between types of content.

NEWSBOTS AND INTELLIGENT AGENTS
Human intelligence doesn't scale to the flow of global information. Informatics tools that represent the interests and intentions of individual human beings will serve as the adaptation that scales human intent to the scope and pace of the new information economy. The ultimate result of a system that incorporates multiple intelligent agents acting on behalf of each individual and organization will be something I've called The Construct, and understanding what is being expressed within The Construct in real time will replace polling, focus groups and market research.

MULTIPLE REVENUE STREAMS AND BUSINESS MODELS
Modern media profits are based on paid content and -- to a far greater extent -- advertising, with an emphasis on display advertising. Future media operations will collect revenue in multiple ways, often receiving a percentage of a transaction whenever its "free" user tools connect buyer and seller. Most advertising will be performance-based and connected to the expressed intent of the user (whether by search or some other function). Traditional display advertising will be a high-end niche for major brands and a low-end function of small-scaled media.

Other news operations will operate as non-profits, receiving no traditional advertising. Some may be run as foundations, or even as informational utilities, governed by boards and bylaws. Much media will be created by individuals and groups that depend on pledge drives to cover their costs, but some of it may be produced by "for-profit social ventures" that blend the power of supply and demand with the intentions of non-profit organizations.

INTELLIGENCE BRIEFING MODEL
One news organization model will rely solely on paid premium subscriptions: News agencies that base their credibility on predictive accuracy (outcome) rather than on "fair and balanced" coverage (process). These services, which treat users as executives to be briefed and prepared, will cost more and will appeal to users who work in highly competitive industries or individuals who make informational awareness a lifestyle choice.

MAINSTREAM RETRENCHMENT
"Mainstream media" today are in decline, with "the people formerly known as the audience" fragmented. Future media will separate into market-driven grades of information. The "mainstream" will become a smaller subset of the total media flow, generally associated with less-sophisticated technology and users who: 1. Produce little content; 2. Profit only marginally from higher grades of information; and 3. Choose a passive lifestyle. Mainstream media will not dominate, but will represent the most significant media plurality.

NICHESTREAMING
Higher-end information users will largely reject 20th century-style mass media, which will remind future users of Stalinesque architecture from the old USSR. These higher-end users will select and manage their personal mediascapes, and media companies will work to connect to these users by identifying and serving many individual niches.

E PLURIBUS UNUM
The current mediascape is built around a recognizable meda "voice" that was established during a period of information scarcity. It is animated by a sense of lingua franca continuity that stretches across newspapers, magazines, TV channels and the academy. The new mediascape will arise from the spirit of unlimited bandwidth and will be fundamentally infused with a limitless diversity of voice, tone and topic. While it will seem a cacophony to older users, new tools and conventions will allow us to experience it on a human scale. From the many will flow a single "media gestalt" -- that we'll experience in many ways.

WATCHMEN WATCHERS
Bias warriors have reduced media criticism to an endless ferreting-out of journalistic hostility toward victimized partisans. In the future, subjective, analog bias hunting will be replaced by a variety of data-driven credibility grades, "best-practices" quality assurances (think ISO 9000 for professional news media organizations), and outside observers. Who is watching The Watchmen? We are. With computers.

CREDIBILITY GRADING
Not all corrections are created equal. Not all lies are as damaging. Stupidity in one sphere doesn't prove stupidity in general. These are the common arguments against the notion that a database approach to credibility grading as a practical application. Yet our current analog system -- based entirely on "the human factor" -- has failed to make useful distinctions on these questions. The reason? Without a system of standards, a human-mediated system cannot respond quickly enough to counter its manipulation by outside parties (Swiftboating). Again: Control doesn't scale.

For credibility to scale to a global information glut, future news media must develop: 1. Systems of publicly grading the "confidence level" of developing information; 2. "Sticky" credibility grades on factual outcomes, both for news organizations AND for news sources; 3. A reputation economy for multiple levels of information users; 4. Transparent processes; 5. Standards-based archiving.

The human element is not endangered, and will account for applying these evolving systems. But future generations will find our fondness for seat-of-the-pants epistemology quaint... if more than a little disturbing.

DEATH OF MONOPOLY PRICING AND PROFITS
While metro newspaper publishers tend to frame the industry's current financial situation as a crisis brought on by declining circulation, the loss of old revenue streams (classifieds) and structural changes in the economy (Big Box retailers vs. locally owed businesses), these statements -- while true -- obscure the obvious. News markets that we once ruled are fracturing into numerous competitors, making it impossible for us to dictate monopoly pricing to advertisers. Newspapers remain profitable, but their profit margins are declining.

The current crisis is a crisis of expectations more than it is a crisis of fundamentals (which are, nonetheless, shakey). Shareholders have come to expect 20 to 30 percent profits from their media holdings, and that simply cannot continue in a diverse 21st century mediascape.

The future of our industry will be based on companies that return profits similar to those experienced in the retail sector. Smaller companies may return higher percentages, but big media will have to learn to get along on single-digit margins.

In a monopoly environment, falling profits mean quality cutbacks. In a competitive environment, companies that choose not to compete for quality choose to die. The winners in the 21st century will be those media companies that make this transition gracefully, fund quality journalism, and learn to be pleased with 8 percent returns.

GAME THEORY
Before video games, electronic entertainment was passive and learning was something that we delivered to young people in ways we determined to be good for them. Not anymore. Today's information users were weened on games that allowed them to explore their environments, and nobody under the age of 60 reads the entire user's manual before diving into a new game. Game concepts -- from user interfaces on news sites to reputation economies on comment threads -- will drive the development of 21st century media. All significant information will be interactive and two-way.

SOCIAL TECHNOLOGY -- VIRTUAL AND OTHERWISE
Social technology extends from the passive (LinkedIn) to the transitory (Twitter), from the
networked (Facebook) to the experiential (Second Life). Social technology can be the delivery platform for news (BREAKING NEWS ALERTS on Twitter), the organizing force behind original reporting (NewAssignment.Net) or the system that sorts and shapes the information stream (Digg). Social technology that incorporates each of these functions will play significant roles in news media within the coming decade for one reason: Social technologies are scalable.

THE WEB IS LOCAL
Earlier this decade I annoyed people by asking this question during high-level discussions of news strategy: Is the Internet local? No one ever said yes.

I meant this question as a challenge to the "local-local-local" insanity that has gripped the newspaper industry, but also as a challenge to think about community in non-geographic terms. And I think I've been proven right: the Internet (and more specifically, the Web) is local, at least in terms of the way people experience their lives.

I've lived in North Central Charleston, S.C., since 2001 and have never attended a neighborhood association meeting. On the other hand, last night I spent hours helping promote the Draft Lessig movement, then donated $50 to a potential candidate for the California 12th congressional district. Why? Because I am more fundamentally a member of the virtual community of values represented by Lessig than I am a member of the North Central neighborhood. I could move to another neighborhood tomorrow, but I would still be the same person once I unpacked.

And why should we disparage online identities relative to meatspace identities? There are more Americans who play World of Warcraft than there are American farmers. If you're trying to be relevant to people's lives, why aren't you covering the kid who made Level 70 in The Burning Crusade with the same degree of interest that you apply to reporting on the kid who won a 4H Award?

Is the Web local? Yes, if by local we mean "of highest personal priority."

Geographically local community coverage will continue to be -- as it is now -- an expensive, high-priority product with a market value capped by geography. Virtual community coverage is also expensive and high-priority -- but its value is limited only by the size and interest of each virtual community. Where would you rather put your money?

TRUE CONVERGENCE
The old idea of "media convergence" meant that newspapers and TV stations would -- in one way or another -- become similar entities on the Web, and news organizations spent plenty of money in the 1990s trying to figure out how to collect 30 percent profits off that idea. The new idea of converged media begins with open-source, structured/semi-structured data streams and flows out to every imaginable form of media, from newspaper to "news games" to virtual worlds to cell phones, via every established 20th century medium (text, still image, audio, video, tabular data, game).

The art of 21st century journalism editing will come in understanding how individual ideas or events are best communicated to target audiences. True convergence isn't about capturing the online video market: It's about learning to surf the wave of constantly churning social and technological change.

Example: YouTube revolutionized Web video in 2005 by offering free hosting for user-created content, plus an essential yet counter-intuitive feature: HTML embed codes. Three years later, most mainstream media have yet to catch up to that advancement because they can't figure out how to think about video within the context of their news operations. That's because they see video as a "thing," just as they see a news story as a "thing." Meanwhile, the video-sharing market is rapidly fracturing into dozens of competing platforms because video is many things. It can be raw, uneven and viral (YouTube), it can be immediate and highly personal (Qik), it can be deliberate and repeatable (BlipTV).

Hence, True Convergence isn't about adding video to your news website. It's about understanding that a breaking news clip of a robbery shot from a bystander's cell phone and a three-minute video story on crime statistics are fundamentally different things. They are only lumped together as "video" in the same way that a limerick and a technical manual may categorized as "text."

True Convergence begins with recognizing the similarities and differences between pieces of content REGARDLESS OF THE MEDIUM THAT TRANSMITS THEM.

CURATING INFORMATION
Wikipedia gets a bad rap in traditional media, typically on the grounds that it is uncontrolled and unfiltered by traditional top-down editorial methods (Reason? Control doesn't scale). While this fundamentally misunderstands the wiki concept, it also ignores entirely the beneficial ways in which people have come to use Wikipedia: As a curated form of search, and as a non-news based method of keeping up with developing information.

Consider: If I'm suddenly interested in what's happening in Kosovo, I can read the news reports on the independence movement. These reports place the emphasis on what's new, since news reporting values novelty. But for me -- since I haven't been paying attention to Kosovo -- that news is an isolated, context-free dataset, and there's only so much context I can get from a 15-inch wire story.

I can go to a traditional, top-down information source (CIA Factbook) to find out more about Kosovo, but I'll have to guess at how current the information is. Or I can go to Wikipedia and read an article about Kosovo that has been edited to included the latest information.

Typical news organizations shun this kind of thinking as "not news. They will soon retire that attitude. Since zapping in and out of topics is the way most informed people acquire information, creating and curating not only databases but high-quality topic articles will be one of the most significant journalism jobs of the future. Again, this will not be instead of news writing, but in addition to newswriting. The best news sources (BBC) already perform this function, often in real time.

NEW ELITES
The Old Elites were economic and institutional, with the occasional popular artist thrown into the mix. While such elites will continue to influence culture, they will be forced to compete with unmediated communities that produce their own elites. Example: Boing Boing's contributors represent an informational elite that is driving culture in ways that elude the control of traditional elites. These new elites will likely appear transitory by traditional standards, but their influence will be profound.

THE CREATIVE MIDDLE CLASS
In the current system, the two options for creative people are rock star or starving artist. In a networked era approaching The Singularity, the creation of content, knowledge and technology will be the primary work of the American economy. For this to function properly, we'll have to develop some kind of stable basis for a creative middle class. Journalists will be members of this class, and would benefit from structures that enable it (health care, new revenue relationships, etc.)

SURPLUS PEOPLE
As development of our knowledge and technology accelerates toward The Singularity, it is likely that the bulk of the human species will become -- in economic terms -- surplus labor. Affordable robots are moving off the assembly line and into our homes and offices, a trend that will accelerate as nanotechnology, green energy and environmentally friendly materials replace the fundamentals of our Peak Oil economy. Media will have an enormous role in dealing with the crisis of unemployable humans, a shift so enormous that we should not fail to include it in our thinking.

YES, NEWSPAPERS ARE GOING AWAY
Not immediately, and not because of TV and the Internet. Newspapers (as we know them) are going away because, as physical products, they are wasteful, create an enormous carbon footprint and pollute our cities. The transition from a Peak Oil economy of waste, consumerism and profitable inefficiency is rapidly moving toward an economy based on life-cycle costs. As regulations begin to require that publishers account for the disposal, cleanup and carbon-mitigation costs of their products, most newspapers will instantly lose real profitability. TMCs will shut down their print editions and switch to Web-only distribution overnight.

The future of newspapers is niche. In 20 years we'll see print newspapers as expensive, "boutique" products for the select few (and the terminally stubborn).

OK, that's what I've got. Think we'll find something to talk about?

Monday, February 18, 2008

"Blackboxing" news judgment

Orson Scott Card's "Ender" series wouldn't be much of a story without a device known as "the ansible," a sort of sub-space radio that allows people to communicate instantaneously from planet to planet across light years of empty space. The story doesn't work without it... but how does it work?

Card's answer? It doesn't matter. The ansible is a black box: In science fiction terms, that means that it has rules, that it abides by those rules, and that so long as those rules are followed, the reader doesn't need to know how it works.

If you get right down to it, that sounds an awful lot like news judgment.

You'll hear a lot of people explain a lot of mysteries as "news judgment" as you go through your career. At its most basic, news judgment determines the difference between a strip "false-lead" and a two-column "news lead" and a bottom of the page "reader." New judgment sorts the daily budget into pages and categories, but it also illuminates decisions about photo placement, cropping, and the tone of headlines.

That's plenty, but we're still not done. Because our profession uses the "news judment" black box to determine more than story placement and layout. News judgment informs our decisions about what stories to cover and what resources they merit. News judgment tells us which voices to trust and which voices to ignore. Information goes in, processed news comes out. What happens in between is... well, as we say: Trust us.

Pause to consider this for a moment. We work in a profession that demands disclosure of interests. We discipline ourselves as reporters and editors to think in rational, restrained ways about competing versions of "truth." We demand that all information have a source, and that we know the source. We set aside what we think and suspect for what we can prove, and if anything in that reporting winds up being even trivially inaccurate, we will fall on our collective swords "because our credibility is all we have."

And at the end of all that work, we turn over all that we do... to a black box.

Is it really a black box? Of course not.

News judgment is one of the arts in what we do. It's supposed to represent the wisdom of our tribal elders, passed down by the generations. It's the pause before taking the bait, the long view in the heat of the moment, the experienced eye that see through the surface spin. News judgment is achieved in part by observing, in part by remembering, in part by reasoning... but it is also largely a function of sitting around talking and worrying. Even when traditional news judgment is done well, it's a messy, fretful process, much more sausage than steak.

But when news judgment is done poorly -- and it often is -- it makes a mockery of those noble intentions. Work in the business long enough and you'll encounter it: Sunday night editors who downplay a big story because they don't want to remake a page on deadline; top editors with hidden agendas they will never voluntarily reveal. Egos and office politics and fear and vengeance."News judgment" is our vague rationalization for all sorts of failures.

One of the cultural shifts that lies ahead of us can be compared to the shift from analog to digital recording. Old LP records -- the kind we listened to when I was a kid -- warped and hissed and popped, but they had a warm sound that digital music doesn't, and there were (and are) still those people who prefer it. But switching to digital music -- each note and quality assigned a digital descriptor instead of an analog wave -- opened doors to new possibilities: CDs, MP3s, downloads, etc.

We stand at that crossroads. There are things I love about our current way of doing things, but then again, I know that the way we do things isn't really a system. Moving ahead into the 21st century is going to require information systems that allow people to make multiple uses of the same data. The past is analog and opaque, the future is digital and transparent, yet we can't budge from this intersection. Why?

Three reasons:

Newsroom culture loves its analog myths: the hard-nosed reporter, the tough city editor, the cynical poet who captures the beautiful ugliness of life in a 20-word news lede, then heads off to the nearest bar to drink himself out of that terrible clarity. We are, so many of us, romantics at heart. Don't turn journalism into a digital representation of data, we say. You'll kill off the human factor -- and that's what really matters.

Then there's another, less romantic reality: We know that, despite all our claims to contrary, we produce a low-grade product. Call it the first draft of history, call it whatever you want, but ask yourself this question: Would you put a guarantee on it? What's the shelf life of what we write? How informed are our decisions? Demanding a systematic accounting for the messy daily miracle that is a newspaper will only reveal how non-systematic we are.

Finally, there's power. Senior editors and the people who influence them have the unchecked power to "make the news" in their own image, and for all their talk, they simply don't want to give that power away.

Twenty-first century journalism will differ from journalism practiced in the 20th century in numerous ways, but I predict the most significant change will be in the way we structure information and account for our decisions. Narrative was our primary tool in the past, but narrative doesn't scale. News judgment worked pretty well when the economics of information were based on scarcity, but it falls apart in an information market based on glut.

Newspaper journalists tell me you can't present the news by formula, that you can't grade information on its confidence. My answer to them? Horseshit. Google News is run by algorithms. Digg is propelled by user input. Intelligence agencies -- one of the models for our 21st century journalistic descendants -- routinely grade the "confidence level" of the information they process.

We're going to need all sorts of digital news products and systems, but none of those developments will matter if they're presented based on some black box called "news judgment." In a world where there's too much information and not enough time, trust will demand transparency and repeatability.

Does that mean we'll no longer have courageous editors who buck the system and do what's right? Or that reporters who tell insightful, moving stories will lose their value? Absolutely not.

Here's what I believe: Once we do journalism in the open, with open-source principles and ethics, we'll have a shot at regaining the credibility we lost over the last 30 years. And once the people learn to trust us because they can test us, they'll be able to see the value of that courageous editor and that insightful reporter.

All they see right now is a black box.

Sunday, February 17, 2008

Gloom and Doom

I hear this quite a bit: Journalists (particularly print journalists) are tired of hearing all the "doom and gloom" about what lies ahead for the industry.

The statement is usually followed by a call for "solutions" and bolstered by high-ranking reassurances that "newspapers aren't going away."

Which probably explains why I'd rather talk about doom and gloom than sweetness and light.

Some points worth remembering:

New communications technologies don't typically exterminate their predecessors (this isn't exactly true: try sending a telegram). Newspapers survived radio, radio survived TV, TV has survived Teh Interwebs, and so on.

Even dinosaurs didn't exactly go extinct. They just stopped being big old dominant lizards and survived as birds -- light, maneuverable, adaptable birds.

Change only looks like gloom and doom when you're living through it.

Hence: The issue isn't newspapers. It's journalism.

If we insist on determining the success or failure of ideas based on the relative successes and failures of newspapers, then we're using the wrong metric. If we insist on valuing only those ideas that return 20 percent profits to investors, then short-sighted greed will blind us to the obvious.

We've got some heavy lifting to do. We need new business models (note that I made this plural) that can fund quality journalism. We need to switch from "document" thinking to "database" thinking. We must re-imagine a relationship with "the people formerly known as the audience" that fundamentally accounts for the fact that the transmission of information is no longer a one-way street.

Will there be newspapers in the future? Sure. Not that most of us will care.

Radio was still around in the 1950s, but Edward R. Murrow -- whose "This is London" Blitz radio reports were the stuff of radio-journalism legend -- had moved on to TV.

So let's stop worrying about preserving the status of the current newsroom management class and the wealth of our media ownership elites.

Let's start scheming up new ways to make great journalism.

Nothing gloomy there.

Wednesday, February 06, 2008

Why quality is a moving target

I got my first regular job at a professional newspaper in my final semester of college. It was a job that no longer exists: Paste-up guy for The Chapel Hill Newspaper.

In those days, editors used pencils to draw page layouts on pieces of paper and stories came out of typesetting machines in long single columns. My job was to take those typeset columns, run them through a machine that coated the back of the sheet with hot wax, and then use an Exacto knife to assemble the stories according to the layout diagrams.

Photos came in pretty much the same way: Editors sized them, the imaging department sent them to a printing machine, and then our crew would cut them out, wax them, crop them, put them on the page and outline them with line tape.

Every once in a while an editor would call for a cut-out photo, and since I had a background as a commercial artist and did a decent job of cutting subjects out of photographs, my skills encouraged more editors to design more pages using more cutouts.

In those days, the definition of a "quality" cutout was a photo that had been cut by someone with a knack for it.

Then Photoshop came along, and suddenly it was possible to do digital cutouts.

FAST FORWARD: PHOTOSHOP
And here's the message: On the day that newspapers gained that ability, the BEST Exacto-knife cutout artist became obsolete. It's simply not possible to do with a knife and a waxing machine what Photoshop users can do with the Magic Wand and various lassos.

New tools -- new technologies -- changed the definition of quality for that specific journalistic convention. The reason was obvious: What used to look good to our pre-Photoshop eyes looked ham-handed once we saw what the new software could do. We understood, without needing to argue over it, that we had to change our standards.

So why is it that when we talk about informational tools, Old School Print Journalists categorically reject the idea that new technologies make our old standards obsolete?

EXAMPLE: ELECTION NIGHT
I spent more than decade running election coverage at two daily newspapers. Having a successful election night is really about logistical planning, which meant that I would typically spend a lot of time thinking about how to process poll results into reliable numbers on deadline. The hard numbers went to "Winner's Boxes," the percentages went to reporters and editors, and if everything ran smoothly, they'd all match-up by final edition.

That was the state of the art in the late 1990s: Dead data in stories and image files, collected over the phone and handwritten by a staff of clerks onto Xeroxed forms that we copied and distributed to be keyed into the system by more clerks. Valuable today, useless tomorrow.

Did we know about spreadsheets and databases back then? Sure. But we had no practical way of integrating those programs into our proprietary content management systems. We couldn't even type the numbers into that system and have them flow into the program that we used to create the winners' boxes.

Then XML came along, and suddenly everyone had the opportunity to build systems that could talk to each other across platforms.

XML allows us to mark up structured or semi-structured data for multiple uses and re-uses. Type it in once, edit it as you would an important story, and then archive when you're done. Write a simple script and the numbers in your winners boxes will update automatically every time you enter new results. Collect enough of it and you can do information magic: Comparisons, charts, searchable products. Combine that data with other tools (the Google Maps API, Flash, Action Script, etc.) and you can create products that were unimaginable when you collected the data in the first place.

That should have made the old "dead-data" system obsolete overnight, just like Photoshop killed all the Exacto cutout artists. But XML has been around since 1998, and most newspapers still aren't using it to manage the data that they process.

Take a look at the cable news network websites: They get it. Cable news spends more on technology because they're working in public in real time. Newspapers don't care, because we work in private and we only show you our final revision. To cable news, technology is a mandatory investment; to newspapers, it's an expensive luxury.

So most cable news sites put up election numbers that are actually being served from databases, rather than typing results from one document into another document. The difference? Change the database and you change all the instances. Without a databased system, you must manually edit each instance that appears on your site.

Because newspapers don't tend to think in terms of databases, they create documents. Even the cool "Interactive Map" that Charleston.net posted after the South Carolina primaries isn't really a true database product. Yes, it nicely mashes up election data to a Flash map, and it looks great. But the data is an isolated capture.

That is to say: because we didn't design the guts of the map around a central database, the data in the map wasn't automatically updated when the state parties provided their final tallies. That's why it's Feb. 6th and our Jan. 26th map still says that 1 percent of Richland County's precincts have yet to report.

To a newspaper editor, that's a quibble. We published. We had a map. It was cool. But I look at that 1 percent and see an Exacto knife cutout. It's a good Exacto knife cutout, but I know that technology has changed the audience's expectations. I know that this change is continuous and accelerating. I know that the data matters.

Why should a newspaper editor care? Because if he'd bitten the bullet on creating an integrated election-results database in 2007, he'd have all sorts of cool things that he could do with with that data in 2012. Or November. Maybe his reporters would play with it and find fascinating stories. Maybe he could open it up to users from the website and they would find fascinating stories. We'll never know.

If you cannot immediately grasp why this is concept is fundamental to understanding the immediate future of 21st century journalism and culture, well... I'll be posting on that later.

Are you thinking, or "quorum sensing?"

In the fall of 2005 I wrote one of my final science-beat articles on research into a biological phenomena called "quorum sensing." Specifically, quorum sensing represents a form of chemical communication between bacteria. That's vaguely interesting, but the exact moment at which quorum sensing transformed my understanding of the world took place when a microbiologist described bacterial behavior as being "kinda like a really bad corporate environment."

Because I realized it wasn't "kinda" like a bad corporate environment -- it was EXACTLY like a bad corporate environment.

QUORUM SENSING IN BIOLOGY
To summarize: Bacteria send out chemical signals into their environment. As they wander about they encounter chemical signals sent out by other bacteria. At low levels of population density, this chemical signaling has zero effect on bacterial behavior, but as a population begins to expand, the background noise of chemical signaling passes a threshold. The result is like flipping a switch: Bacteria that acted one way below the threshold suddenly behave in radically different ways. And they do so like a light turned on by a switch rather than like a light turned on by a rheostat.

Hence: When an individual bacteria receives only a few chemical signals, it acts as an individual. But once it perceives that there are sufficient members of its species in the area, it cooperates in complex ways to create a colony. Once that colony is established, quorum-sensing bacteria will serve the colony even if that means committing suicide.

This is why having one or two e. coli on your hamburger won't make you sick. Individual e. coli are waiting for a signal to start acting aggressively. A few e. coli acting up would immediately be overwhelmed by your immune system; an army of e. coli, conducting a coordinated surprise attack, will put you right on your ass.

(Interestingly, this also takes place in the animal kingdom. Ever notice how fire ants tend to bite all at once? That's because they're using chemical signals. If the first fire ant to reach your ankle bit you right away, you'd kill it and brush off the rest. Instead, the ants wait for the signal that says a bunch of them are in position, and then they bite simultaneously. It's an evolved a form of communication that enables ants to inflict maximum damage on their enemies.)

And how do bacteria colonies compete for resources? Our classical, Western, market-based idea of competition holds that whichever competitor is the most productive wins. But quorum-sensing bacteria often win by cheating, secreting chemicals that poison or confuse their competitors.

Does the best bacteria colony win? Not usually. The most organized and established bacteria win.

And since our subject is media change, this point is significant, too: In a mature bacterial ecosystem, there is little actual competition. All the organisms in the ecosystem have evolved to exploit their own niches. What sets off bacterial war is anything that upsets that equilibrium. Competition -- often violent -- occurs whenever nature encounters a vacuum.

QUORUM SENSING IN THE MEDIA
I believe it helps to think of mass-media as an ecosystem. Newspapers and TV stations competed for scoops, eyeballs and ad revenue, but TV didn't threaten to put newspapers out of business, and vice versa. That particular media ecosystem remained more or less stable from the early 1960s to the early 2000s. The World Wide Web began threatening it in 1994-95, but it took the mass distribution of broadband access and a series of subtle technological advancements to actually disrupt the equilibrium that pays journalists' salaries.

We practice journalism today in the transitional period between an old equilibrium that has ended and a new equilibrium that has yet to take shape. The outcome cannot yet be reliably predicted, and the notion that the best, most productive ideas will naturally rise to the top is far from proven.

As I look around, I see a lot of companies acting like bacteria colonies. They send out signals and try small initiatives, but few are moving in any bold, wise directions. Many executives are just sitting around, receiving signals from their environment, waiting for the signal that a "quorum" has coalesced around a new direction.

In other words, their actions will not be determined by an independent, forward-thinking assessment of individual ideas, but by their perceptions of where everyone else is going.

On the one hand, this isn't a bad approach. It's certainly traditional, and it certainly offers at least the illusion of safety. But this corporate quorum sensing has also been the cause of some amazingly foolish industry fads -- like the "pay-to-read-our-website" push and the "heavy registration" mandate. It's not like the experts didn't tell the executives these were bad ideas at the time -- it's just that everyone understood from the signals in their boardroom environments that the CEOs and shareholders were tired of websites that didn't make profits.

QUORUM SENSING WITHIN THE PROFESSION
Journalists like to believe that we are -- as a profession -- a tribe of free-thinking individualists. It turns out we're not that different from other professions, with the same incentives toward group-think, quorum-sensing and anti-intellectualism.

It's also time to call bullshit on the newsroom tendency to imagine that all these conformist, bottom-line tendencies arise with the beancounters and are then forced upon our noble First Amendment enterprise. Newsrooms are, by their very nature, conservative institutions that abhor anything that disrupts the production cycle. We're actually hostile to innovation.

You'd think that an entity that produces something new every day would be adapted to rapid change. The opposite is true. A newspaper is a physical object that is printed and delivered at the same time, every day. Only what goes into it changes. and so long as what goes in doesn't upset the production process or change the physical object, content barely matters.

But try to change any of those variables -- deadlines, workflows, meeting schedules, the relationships between our print products and our electronic products -- and watch all hell break loose. A newspaper is a machine honed to perfection by time. It can't adapt easily to new things because it's become so efficient at doing the same thing, over and over.

Why bother to talk about this? Because as we discuss creating or improving quality in the digital environment, what we're really talking about is re-engineering the entire environment. Print journalists want to talk about saving newspapers, and that's the wrong topic. The real issue is how we'll change journalism to function with 21st century tools.

Why talk about this in the context of bacteria? Because print journalists by and large still discuss new-media journalism based not on experience or study, but on their quorum-sensing perception of their peers' attitudes. New media tools like database/map mashups represent wonderful new opportunities for advancing the original goals of journalism, yet print journalists still tend to frame their internal discussions of new media as a narrative about civilized people besieged by barbarians. These men and women are not thinking about the future, they are conforming to their perception of the present.

And if you are outside of that perceived consensus, you're likely to be very lonely.

PUTTING QUORUM SENSING TO WORK FOR YOU
A shorter version of this post might be: "Being right is less important than being normal." Because something about human psychology tells us that the more often we hear or see something in a non-threatening way, the more normal it becomes.

Two things will have to happen before we'll succeed in creating quality journalism for the new media ecosystem. Thing No. 1 is we'll need a functional business model; Thing No. 2 is we'll need to convince everyone in the business that our ideas fit within the mainstream.

For instance, I predicted two years ago that the future of news reporting will include geo-tags for every location we mention in stories. And the technology exists today to integrate this into the newswriting workflow. The costs of including this in our reporting are minimal, the benefits are significant, and you could make money by doing it.

So why aren't newspapers doing this?

Because it seems weird. Because it's new. Because the people who make these decisions can't quite imagine the new products that would make use of this data or the process that would create a geo-tagging interface for newswriters and editors.

But here's what will happen: As executives are exposed to more geo-data mashups, they'll begin to perceive that geo-tagging is OK. Geo-tagging will start to seem like a normal thing to do with bits of information. They'll start to see the profit potential in it, because companies like Google will be making money off of it.

And on that day, some newspaper executive will ask his assembled subordinates,"Why aren't we geo-tagging all our stories? Why are we behind the curve again!?"

In other words, seeing the future isn't enough. We have to communicate that future over and over, spreading our good ideas and winnowing out our bad ideas. We have to understand that our decision-makers will have to see our ideas crop up in many places before they'll see them as valid.

News veterans -- and I'm one of them -- are not generally enamored of the newest generation of entry-level reporters. We question your work ethic, your willingness to learn, your willingness to pay your dues. I think there are lots of things young journalists should learn from veterans -- but let's be frank: You're also in a position to help us advance new ideas about journalism and media.

You can help by leading, but you can also help by treating new ideas as normal evolutions of old values. You can become quorum-sensing transmitters of normalcy to anxious news executives. You can think about novel technologies without feeling threatened by them. You can see change as an opportunity rather than a threat, and you can communicate that with your attitudes.

Whatever the fate of your generation, you can play an important role in the current transformation from the old equilibrium to the new. It might even make some of you stars.

"Charlie, here comes the deuce. And when you speak of me, speak well." -- Crash Davis, Bull Durham (1988)

Tuesday, February 05, 2008

Quality and other essential bullshit

"The place to improve the world is first in one's own heart and head and hands, and then to work outward from there. Other people can talk about how to expand the destiny of mankind. I just want to talk about how to fix a motorcycle. I think that what I have to say has more lasting value." -- Robert Pirsig, Zen and the Art of Motorcycle Maintenance, 1974
Our topic for Feb. 25th (absent a spiffy name-change) is "Quality in the digital environment." And obviously, with professional journalists coming in to talk to journalism students and faculty, the form of quality we're going to be discussing is ...

Musical quality.

Or we might as well be. Because "quality" turns out to be one of those philosophical topics that professional journalists simply hate. So to get things started, let me encourage you to begin your thinking about journalistic quality in the digital environment by forgetting about the journalism and the vaguely defined digital environment and focusing on that elusive word: quality.

So: Back to music.

You know good music when you hear it, right? And your tastes in music have improved with age, as you've listened to more music? Most likely.

But if I asked you to define -- in terms that apply to all music and to all listeners, in all meaningful situations -- what makes one piece of music quality and another pure crap, would you consider that a valid request? Where would you begin?

And if it's not possible to define a universal standard of quality for something as universal as music... well, how are we going to do that trick for journalism?

We all think we know quality when we see it (or hear it), but when we attempt to define what quality is, we can't get there. We can't pin quality down, measure its absolute terms, cite its sources. In touching upon this word, we are entering into hostile territory: the realm of the philosopher.

Quality is, at its heart, a matter of metaphysics, and journalism -- as a set of tools -- is poorly equipped to deal with metaphysics. Our profession is a mental discipline in which we are asked to weigh multiple bits of information and perspective and then responsibly reason our way through conflicting possibilities by asking: How do you know? Where is your proof? In our business, the material always trumps the intangible: You've got an anonymous quote? My on-the-record quote trumps it (regardless of the quality of the two quotes, by the way). You've got three people who'll say something on the record? My official, FOIA-obtained document kicks their collective asses. And so on.

So when you try to talk philosophy with journalists, be prepared. Our brains short-circuit. What journalists really believe is that things that cannot be sourced in a materialistic way are -- in essence -- irrelevant. Which means that quality is just a word, and people who say things like "quality cannot be defined" are simply eggheaded bullshit artists.

OK. Got it.

But understand this: To be great, you must be willing to tackle the intangible. To build better journalism, you simply must struggle with the abstract dilemmas of quality, because the answers you find will point you to new possibilities.

Are all opinions of quality equal? Should decisions about quality be trusted to the people or to informed elites? What expertise improves our understanding of quality? Can quality be measured by popularity? Or profitability? Should we measure journalistic quality by how closely we hew to abstract notions like "truth," or by tangible metrics, like correction counts? Is quality found in a process or discipline, or does it reside solely in the end product, however created?

Why should anyone care about this? Because brand new tools are changing the context in which we commit journalism. My charge to you: Do not judge new tools by old assumptions -- particularly those that reject the validity of abstract criticism.

Because we do not look to improve the quality of journalism for journalists. Or to make more money for stockholders. And we certainly don't do it to validate the grumpy opinions of our high priests.

The only reason to improve the quality of journalism is to serve people. And we should get about it.
"Any effort that has self-glorification as its final endpoint is bound to end in disaster." -- Robert Pirsig, Zen and the Art of Motorcycle Maintenance.

Wake up!

I mothballed this blog in 2007 for many reasons. Some of them were conceptual (I felt that discussing media absent the larger context of culture was pointless); others were personal (I'd stepped down from a "management" level job and returned to reporting).

But it occurred to me last week that it was time to bring it back in a new, and very specific, incarnation: From now until Feb. 24, I'll be using this site to prepare students at the University of Mississippi for my visit on Feb. 25th. Janet and I will be appearing on a panel with the tentative title "Quality in the Digital Environment," then sticking around to talk with the staff of the student newspaper there.

There's a great story about a radio talk show host and fan of Noam Chomsky who once ambushed Jeff Greenfield about why Nightline wouldn't invite Chomsky on as a guest. Greenfield's answer -- oft-derided but certainly candid -- was to ask a question: Could Chomsky condense his ideas down to 30-second answers?

And obviously, the answer is no. You can't introduce an idea in 30 seconds. All you can really do in 30 seconds is reinforce existing ideas, scoring shallow debating points against "the opposition."
I have no interest in boring people who don't care about my topic with lengthy explanations. In fact, I really have no interest in people who don't care about my topic -- which, based on experience, will likely be most of the people in the audience. But I'm intensely interested in the self-selecting few who will find something of value in the ideas I represent.

To those people, I dedicate this new incarnation of this old blog. I'll try to post a steady stream of ideas and concepts here, then leave it around for reference for anyone who comes along later.