Stupor Crunchers

A friend of mine, Steve Fiore at the University of Central Florida, tried to ruffle my feathers by sending me a review of a book by Ian Ayres Super Crunchers. The takeaway message from Ayres’ book is that statistical analyses are more trustworthy than expert opinion.

Ayres is an econometrician and law professor at Yale, and his book describes how data mining is replacing expertise and intuition. In fact, the alternative title for Ayres’ book was “The End of Intuition,” but when he test marketed titles by running Google ads, “Super Crunchers” won out.

Ayres describes different examples of data mining and shows how this trend is increasing – to give more authority to central computers rather than to people, particular people in clerical positions. Quantitative analyses are even outperforming expert wine tasters. Almost 20 years ago Orley Ashenfelter derived a simple formula based on weather data to accurately predict which years would produce the best wines. Wine experts were contemptuous and outraged, but Ashenfelter’s formula seems to do a better job than they do. (I suspect that Ashenfelter didn’t examine every possible variable to see which correlated with wine quality but used existing expertise to nominate the most likely variables.)

I’ve been reading these kinds of polemics about analysis outperforming expertise/intuition for years and they strike me as silly, like the fantasy tournaments children sometimes invent: Can a lion beat a shark? (It might depend on whether they were fighting above or below the waterline.) Who would win, a superhero with infinite strength or one with infinite cunning? (It might depend on the rules of the contest and the resources available to the contestants.)

Yes, you can achieve surprising insights with statistical analysis. But the analytical methods work best in well-ordered situations. When you move to complex and chaotic situations the analyses often become brittle. That is, when you have lots of moving parts that are interconnected in different ways, with dependencies that depend on context, particularly on a changing context, then the analyses become less reliable. I think Mr. Snowden may have suggested something along these lines.

Books like “Super Crunchers” (and, I admit, I haven’t actually read this book, only two reviews of it so I hope I am not being unfair to Dr. Ayres. I have ordered a copy of the book but I won’t have a chance to read it before I am done blogging for the Cognitive Edge website) only invite rebuttals showing all the damage created by quantitative analysis (e.g., my last blog on the collapse of the sub-prime mortgage market), plus examples of the amazing things experts can do. A mindless reliance on quantitative methods turns people into stupor crunchers capable of making large blunders by being precisely wrong.

Case in point: the hit TV show “Everyone Loves Raymond.” Careful analysis of market research data led the studio executives to pass on this show but one of them, an advocate, felt that the data were misleading and gave it a green light. It went on to generate $1B in profits during the next 10 years.

And I recall that television executives also drew on data to decide not to pick up a music show from the U.K. The three major studios all passed on this show and Fox was also getting ready to pass on it until the studio head spoke to his daughter who was living in England and who convinced him to take a chance on it. I believe its U.S. version is called “American Idol.” Of course, we can also find examples where the intuitions of the Hollywood executives resulted in hugely expensive fiascos. And that’s the point. When asked which you trust, intuition or analysis, the correct answer is: neither.

I had hoped that we would have moved on by now. The system 1/system 2 formulation that Danny Kahneman and others have espoused seems to represent a nice middle ground (although there is some suggestion that system 2 needs to be further subdivided). We need expertise/intuition and we need deliberate analysis. Both are valuable. Neither is sufficient. End of story. Can we all be friends now?