Speaking of Silver’s FiveThirtyEight, preliminary signs are not good. That’s unfortunate. As someone with a scientific background who is often frustrated with mainstream journalism’s tendency to abdicate responsibility for determining the truth of anything, the pitch sounds great. Silver, who gained notoriety as a statistician able to punch through conventional wisdom first in the realm of sports, then in elections, by focusing on the numbers, presents his new site as a venue to expand that approach with more staff able to cover more topics.

Focusing on numbers seems like a great way to go. Any field gathers its share of Very Serious People who know what they know and know that they’re right because, after all, they’re successful, how could they be wrong? And when they aren’t successful? Well, they were only saying what everyone else knew, too. No one serious disagreed. You can’t be mad at them without being mad at yourself. The media loves talking to these people because it’s easy. It turns out that understanding every bit of knowledge you might need to come across in the course of reporting is hard. Whereby “hard” I mean “impossible.” One easy, and better, safe, way out of this is to punt. Contact a successful, respected person and report what they say. If that position is deemed too controversial, just find another who disagrees and print both. We are rightly hungry for journalism that takes responsibility for vetting its sources and is willing to call bullshit when someone states what everyone thinks they know, but the data disproves.

Unfortunately two people I trust a great deal when it comes to using numbers to draw conclusions about the world, aren’t impressed so far. Sam Wang,¹ like Silver, has successfully tackled the task of predicting election results via aggregated polls over the last few rounds at the Princeton Election Consortium. He points out via Twitter that, “stat-worshippers are also hedgehogs.” This is a reference to Silver’s stated goal for FiverThirtyEight to be a “fox” not a “hedgehog.”²

But numbers aren’t just numbers. Interpretation matters. Knowing which numbers to chose matters. And the way you move from numbers to a conclusion matters. Paul Krugman pointed this out on his blog:

But you can’t be an effective fox just by letting the data speak for itself — because it never does. You use data to inform your analysis, you let it tell you that your pet hypothesis is wrong, but data are never a substitute for hard thinking. If you think the data are speaking for themselves, what you’re really doing is implicit theorizing, which is a really bad idea (because you can’t test your assumptions if you don’t even know what you’re assuming.)

I sincerely hope they get things turned around. We desperately need data driven, opinionated reporting coming from sources high profile enough to shift the conversation. But we run the risk of being even worse off than we were with bad appeals to authority if we believe that numbers immunize us from other sorts of critical thinking. There’s a reason why statistics and lying are so connected in popular imagination.

Update: Again via Sam Wang, some more specifics on the early missteps of the FiveThirtyEight crew. The Way Things Break covers some of the same ground from above, then dives into their misuse of statistics and flirtation with climate change denial via the hire of Roger Pielke Jr., who apparently has a bad habit of using statistics to argue a predetermined point while getting much of the basics wrong. Paul Raeburn at MIT’s Knight Science Journalism Tracker blog covers how Jeff Leek, writing for FiveThirtyEight, dresses up an exercise in garbage in, garbage out and, by cloaking it in numbers, disguises it as a statistical approach to determining validity of a health news headline.