Separating signal from noise on Twitter

Results of a Twitter search on earthquake near:"Los Angeles" within:15mi

By now a lot of people in the media have discovered how to use Twitter as a promotional tool, judging from the growing number of auto-generated messages populating (polluting?) the Tweetstream.

But I think relatively few journalists are actually listening to what the community is saying. Which is a shame, because this is our audience talking. And the conversation is often more transparent, more sincere and more insightful than what you see on news sites’ forums and comment boards.

(I should note that I’m a relative Twitter novice, and I welcome the opportunity to get schooled if I’m totally off base in what I’m about to say.)

Steve Yelvington (who, incidentally, was helming when I was an intern there more than a decade ago), describes Twitter this way:

It’s like a big caffeine party. Everybody’s talking at once. Really fast.

But you have magic ears.

You only hear the people you want to listen to, and the people who are saying something directly to you.

That is Twitter’s great promise, but it’s also where I think the microblogging behemoth comes up short. Because two things happen when you’re listening only to the people you want to hear:

  • They say a lot of things you don’t care about.
  • You miss all the good stuff they’re not talking about.

So, how can journalists separate the useful stuff from the chatter on Twitter? There are some technological answers to this question. Here are a few I’ve found:

  • Twitter advanced search: Sure, everybody knows about Twitter search, but the advanced search options can be a pretty effective way to cut through the noise. For a simple example, try this: earthquake near:”Los Angeles” within:15mi (The geographic search makes use of the place name that users set in their profiles as a geotag for each of their tweets. Imperfect, but it’s a start.)
  • TwitScoop: See what’s trending in real time. In the future, I imagine a local version of this (limited to tweets in a particular geographic area) on a big display on the wall of every newsroom.
  • TweetDeck: A beautiful app (built with Adobe Air, for good compatibility karma) that lets you follow multiple subsets of the tweetstream (your replies, custom searches, etc.) in real time. If you cover a beat, why not set up a few custom searches on the topics you follow and see what people are saying?

And then there are some things I wish I could do with Twitter that, as far as I know, aren’t possible yet. Here’s this journalist’s wish list for Twitter and third-party developers:

  • Better geographic tools, so it’s easier for tweeters to update their location and for searchers to filter geographically. Community news sites could benefit from this.
  • The ability to create running searches across a subset of Twitter users. Let’s say you cover technology and are following a few key sources and want to know whenever one of your sources posts about Yahoo. There might be a tool that enables this, and if so, maybe somebody can enlighten me.
  • The ability to find conversations that mention a particular URL. Seems like it would be useful (and not just for ego purposes) to know what people are saying about the content you create. Twitturly does a good job of showing which URLs are most popular overall, but as far as I can tell it doesn’t let you specify a URL to examine.

So, what are your techniques for separating signal from noise on Twitter? And while you’re at it, what would you add to the wish list?

Incidentally, if you’re new to the Twitter thing, here are some good posts to get you going. And for a contrarian point of view on the whole signal-to-noise thing, check out Scoble. (Or, maybe he’s mainstream and I’m the contrarian? [Shudder])

Making sense of data at The New York Times

(After a long holiday hiatus, I’m finally getting around to posting this write-up of my visit with Aron Pilhofer at the NYT.)

"Movable Type" at The New York Times building

Tuesday, Dec. 23, 2008: The digital art installation in the lobby of the new New York Times building says more, I think, about the future of news and of the Times Company than its creators may have intended. Yes, we know that the future is digital and real-time and kinetic, like the work by Mark Hansen and Ben Rubin. But, more than that, the journalism of the future will be defined by its capacity to extract meaning from countless bits of data. The work, titled Movable Type, elegantly illustrates the bits. Making sense of them is Aron Pilhofer‘s domain.

It is my first visit to the new building, directly across 8th Avenue from the Port Authority Bus Terminal. I am meeting Pilhofer, who leads the paper’s interactive news technology team, for a quick tour and chat. His group of 10 developers, assembled over the last year or so, works on editorial projects (such as the Times’ live election results) but doesn’t report to the newsroom. Their boss is Marc Frons, the website’s CTO.

Over cups of caffeinated liquid in the Times’ airy 14th-floor cafeteria, Pilhofer tells me about Represent, a newly launched project from his team that, as the name suggests, lets you “keep track of what the people who represent you are doing.” Though still in soft launch, it’s already generating some nice buzz. (A bit on the tech specs from co-creator Derek Willis: They’re using GeoDjango to drive the mapping features.)

Pilhofer is an archetypal journo-techie, raised in the computer-assisted reporting school and fluent both in the cadences of the newsroom and in the technical lingo used by his fellow geeks. Before joining the Times’ computer-assisted reporting team, he honed his skills at the Center for Public Integrity, a D.C. nonprofit that seems to have been a sort of proving ground for smart, webby CAR folks (Willis and my former LAT colleague, Ben Welsh, are also alums).

This is the breed of journalist that web-oriented newsrooms would like to find more of. The problem is, “they just don’t exist,” Pilhofer says of his ilk. When I throw out the old question about whether it’s easier to teach a journalist programming skills or to teach a techie the principles of journalism, he tells me it’s not so much a question of trainability. Rather, he says, “there are more programmers out there that will find journalism interesting to learn” than vice-versa. He tells me that, with a couple of exceptions, the people on his team have either “very limited journalism experience or none whatsoever.”

Given that most of Pilhofer’s group comes from a hardcore tech background, I wonder whether they’ve acceded to rigid product development conventions like wireframes and detailed requirements documents. His response: “Hell no.” (Actually, he uses a more colorful four-letter word, but you get the point.)

He does throw out a lot of prod-dev terms like agile development, scrums, Extreme Programming and pair programming, but he uses newsroom analogies to describe them. Agile development methodology, for example, which stresses frequent deadlines and shuns long meetings, has a lot in common with the rhythm of a newsroom. And pair programming, an unconventional workflow in which two coders work in tandem on the same problem and test each other’s work as they go, is analogous to team reporting.

Some other highlights from our chat:

Software: Pilhofer says his team relies heavily on open-source solutions. Ruby on Rails is the workhorse in this shop, but it’s been adapted to produce flat files when necessary (as opposed to rendering pages on the fly), a performance tweak that enabled the Times to keep up with unprecedented traffic to its election results data.

Hosting: Amazon’s EC2 service, used for most of the team’s data projects, has enabled them to scale with demand. “Amazon has been the savior of this group,” Pilhofer says.

Newsroom geography: The interactive news technology group sits on the Times building’s second floor, in close proximity to the graphics and CAR teams, the two groups Pilhofer says his team works most closely with. (The paper’s business desk takes up much of the rest of the floor.) The graphics desk, in particular, has been a close collaborator, bringing sophisticated visual interpretations to many of the team’s projects. Pilhofer calls deputy graphics director Matthew Ericson the “de facto co-manager” of the interactive news technology team.

Roles and hierarchy: Responsibility for the Times’ interactive projects is shared among Pilhofer’s team, the graphics department and other groups in the newsroom (a highly collaborative, loosely organized structure that reminds me of how interactive projects got done at the L.A. Times, but on a much larger scale). “I kind of like the way it’s working right now, where there isn’t some big, centralized, one-person-in-charge-of-everything,” Pilhofer says. “I think it’s healthier.” Each group brings certain strengths. For instance, the graphics folks want to do really intense, deep immersive online interactives, but they can’t do that without back-end help from Pilhofer’s team, so the two groups work together. Organizationally, Pilhofer says his team benefits from a direct connection to the website’s software and infrastructure folks while other teams are more closely tied to the newsroom. The downside to this setup, of course, is that it’s sometimes hard to know who owns what.

More Pilhofer: Old Media, New Tricks recently published an interview with Pilhofer.

Update 2009.01.13: Emily Nussbaum has a feature on Pilhofer, Ericson and other NYT geeks in New York Magazine.

Coming next week: A look at DocumentCloud, a promising Knight News Challenge proposal from Pilhofer and ProPublica’s Eric Umansky and Scott Klein.

Photo by Eric Ulken.

Ironic flashback photo of the day

YOU Own This Place Now!

Above: Los Angeles Times employee entrance, Dec. 21, 2007. Those signs seemed a bit disingenuous even then, given the structure of the deal to take the Tribune Company private. Now they seem, well, ridiculous.

Update: Ex-Daily News ed Ron Kaye’s thoughtful reaction to TribCo’s bankruptcy filing: It’s the beginning of the end for newspapers, but it’s also a great time to be a consumer/creator of news online. Not sure I’m quite as pessimistic as he is on the first point, but I totally agree with him on the second. (Hat tip: @busblog)

A first draft of the itinerary

Newspaper boat

I’m busy packing up my L.A. apartment in preparation for my upcoming adventure. So I haven’t had much time to spend on travel planning, but things are starting to take shape. Everything is subject to change, of course; here’s what it looks like now:

  • Europe: I will be there roughly from mid-January to early-April. It’s a work/play trip; I’ll be visiting family and friends in addition to my reporting. I plan to spend most of February in the U.K. Otherwise I’ll hang out on the continent, checking in on individuals and organizations doing innovative work that’s relevant to the practice of online journalism. I’ll be blogging the results here (and possibly on other websites, though everything will be linked here).
  • South Africa: I’m hoping to tack a couple of weeks there onto my time in Europe. I’m interested in looking at how the digital revolution is unfolding in societies with large unwired populations. Are cheap and ubiquitous mobile phones taking the place of PCs as news delivery platforms?
  • U.S.*: In late April and early May I’ll do a little reporting back in this country. I plan to focus on trends in database apps and information visualization for news. (*Update: I overlooked Canada here — an unintended slight as I do hope to make it to Vancouver at least.)
  • Latin America: In May I will try and get to Mexico and/or South America for a couple weeks. This is still pretty iffy.
  • Japan: I plan to head to Japan in June or July. I promised my teenage brother, a budding Japanophile, that I’d take him so he can practice the Japanese he’s been learning (and, hopefully, be my interpreter). I’ll use the opportunity to investigate this apparent paradox: How is it that one of the world’s most wired (and wireless) societies is home to seven of the 10 largest newspapers on the planet?

Why aren’t Australia/China/India/the Mideast/etc. on the list? There are lots of great stories in each of those places, I’m sure, but I have to draw the line somewhere. As it is, I have no idea if I’ll actually get to all the places I’m shooting for. Either way, by August I will be thoroughly tired of traveling and writing, and I’ll probably go off the grid for a while. 🙂

I’ll post more details as I figure them out. If you have suggestions for places to visit or people to contact — or you can hook me up with a tour guide or a place to stay in one of the stops on my itinerary — holler.

Update 2009.01.24: An up-to-date itinerary can be found here.

Photo by marcelgermain via Flickr.

My next assignment: covering online journalism

I’ve spent 10-plus years working from within to change newspapers in some small way. Now I hope to effect change from the outside. Earlier this month, I left my job as interactive technology editor at the Los Angeles Times to travel and learn and share stories about the great work taking place in online journalism around the world. I love the Times, my work and my colleagues, but I’ve decided it’s time to try something new: reporting.

Beginning in January, my plan is to spend six months or so writing about trends and best practices in the field, both in the U.S. and abroad. Among the questions I’d like to explore:

  • What common themes emerge as news organizations change their workflow, culture, reporting structures and newsroom geography?
  • What are newsrooms in Europe and elsewhere doing that American media can learn from?
  • How are news organizations succeeding in doing more with less? Where are they focusing their resources and what are they walking away from?
  • What lessons can we take from success stories outside traditional media, including solo practitioners and online-only outfits?
  • How are news aggregators and social media affecting coverage choices and marketing of content?
  • What new storytelling and data presentation forms are gaining traction? How are viewers reacting?
  • How are traditional media companies altering (or blowing up) their business models to compete in the new information economy?
  • What role are the people formerly known as the audience playing in the newsgathering process?

Kind of broad, yes, but I’ve always been more of a generalist than a specialist.

A lot of people are blogging their opinions about the state and the future of journalism. I have plenty of my own, and I’ll share them when I think they’re relevant. But mostly I want this to be a fact-finding mission. I am not a reporter, but that is what I’ll try to be for these few months.

I believe that much of the journalism newspapers do is still important and essential, and I want to see that work live on after print dies. So I intend to write with an eye toward helping traditional news organizations negotiate the terrain of online media, but I hope that some of these topics will be of interest to people beyond “old media”.

I’ll be blogging here at, unless some generous benefactor agrees to finance all or part of this endeavor, in which case I’ll write wherever I’m asked to.

I’m looking for guidance on where to go, who to talk to and what topics to investigate. Please leave your advice in the comments here. I plan to base my itinerary in large part on the suggestions I receive.

More soon.

Update 2008.12.05: Thanks for all the great thoughts. A few more details on the itinerary. Also, ran a little e-mail interview on my plans.

Update 2009.01.24: An up-to-date itinerary can be found here.

Who says newspapers are dead?

The scene at Second and Spring streets, downtown Los Angeles, 2 p.m. PST Nov. 6, 2008:

For the second day since the election, the line of people seeking copies of Tuesday’s paper outside the Los Angeles Times building is around the corner. (Copies of the paper are also listed at a substantial premium on Ebay.)

Jack Klunder, the paper’s president, told me the Times printed a “couple hundred thousand” more papers to satisfy the demand. It’s also selling plates of the paper’s front page for $10 a piece and lithographs for $5. The Times’ Reader’s Rep blog has details.

(Update 11/9: Sandy Banks has a nice column, with an accompanying Sachi Cunningham video, on the phenomenon. When I walked by the Times yesterday the line was still out the door and some enterprising independent vendors had set up shop across the street selling Obama T-shirts and other merchandise.)

Finally, for those who prefer the digital version, here’s a photo I took in the newsroom Tuesday night:

New at Slice and filter California election results

In my last few weeks at the Times, I’ve largely been preoccupied with imagining and building our election data widgets for use on election night.  It might seem silly to spend so much time preparing for an event that’s over so quickly.  But I think we’ve found at least one app that’ll last long enough to make it worth the effort.  It’s our California county-by-county map, and I think it’s way cool.

Sure, you can see bubbles by county for the state propositions and the presidential race, but you can also slice the vote by demographic categories (e.g.., counties that went for Bush in 2004) to see if you can spot trends. Happy filtering.

(Credit where it’s due: The filters are inspired by NYT’s excellent county-by-county maps during the primary season.)

Leaving the Times

I want to say something about what took place today at the Los Angeles Times, where I’ve worked for nearly 5 years. It’s a drama that repeats itself in newsrooms across the country and has already taken place more times than I care to count during my tenure at the Times. The familiarity of the event doesn’t make it any less sad.

I refer, of course, to staff cuts. Buyouts, redundancies, layoffs, terminations, separations voluntary and involuntary — pick your term. However you put it, it sucks — both for the people who leave and those who stay. This time I’m in the former category. The decision to go was mine, and I made it months ago, but saying goodbye is still hard.

Days like today obscure the fact that hundreds of talented, creative and dedicated journalists remain at the Times. They still put out one of the best news reports in the country, and they’re working hard to drag a sclerotic institution into the digital age. I am proud to be associated with them, and I wish them well.

(My last day is next Friday, Nov. 7.  Details on my own plans TK.)

Photo by Mister-E via Flickr.

Backchannel on blogging

What happens when you put a bunch of bloggers in a room, feed them pizza and moderate a discussion on their craft? You end up with two real-time conversations: One in the physical room and the other in the Twitterverse.

I know that’s no surprise to those who populate this corner of cyberspace, but as a newbie here — and, until recently, an admitted Twitter skeptic — I have to say it was pretty fun to watch a virtual dialogue unfold alongside the real-world one, as it did last Thursday night at the Los Angeles Times.

The local ONA gathering on blogging (pix here) drew about 60 people to the Harry Chandler Auditorium for an informal talk with L.A. bloggers including Luke Ford (pictured) and the Times’ Andrew Malcolm. Tweets containing #onala were displayed in a search feed on the big screen, powered by a nifty skinnable Twitter client called Spaz. Instant visual backchannel!

So yes: Twitter is good for something besides marriage proposals.

Photo by David LaFontaine via Flickr.