Prediction failure

Follow-up to Probability neglect from Autology: John Dale's blog

I was thinking yesterday about how people aren't very good at considering probability when they think about future events, and coincidentally today I've been reading about some pleasing research showing that people – especially "experts" – are in fact generally pretty rubbish at prediction generally.

This review in the New Yorker of Philip Tetlock’s new book, Expert Political Judgment: How Good Is It? How Can We Know? talks about the surprisingly large number of areas in which experts on a subject do no better in the accuracy of their predictions than people who know nothing about the subject, or even blind chance.

(As an aside, I love the New Yorker. Pretty much everyone who writes in it manages to be entertaining and witty, and for reasons I don't fully understand, they also seem to consistently write about things I find interesting. Malcolm Gladwell's articles are invariably great (must write about him in a separate entry at some point), and Anthony Lane is an insightful and funny film critic: talking about the new Narnia movie, he says "The one thing delaying any attempt to film the Narnia novels was the lack of technology; until recently, for example, there was no computer-imaging program powerful enough to re-create a wholly convincing wardrobe". I wonder how much international subscriptions cost?)

Where was I? Oh yes; experts are rubbish at predictions. Philip Tetlock's book is full of striking examples: data from a test used to diagnose brain damage were given to a group of clinical psychologists and their secretaries. The psychologists’ diagnoses were no better than the secretaries’. Political forecasters asked to choose between three predictive alternatives did worse than chance. And students were bested by lab rats because they over-thought the problem:-

A rat was put in a T-shaped maze. Food was placed in either the right or the left transept of the T in a random sequence such that, over the long run, the food was on the left sixty per cent of the time and on the right forty per cent. Neither the students nor (needless to say) the rat was told these frequencies. The students were asked to predict on which side of the T the food would appear each time. The rat eventually figured out that the food was on the left side more often than the right, and it therefore nearly always went to the left, scoring roughly sixty per cent — D, but a passing grade. The students looked for patterns of left-right placement, and ended up scoring only fifty-two per cent, an F. The rat, having no reputation to begin with, was not embarrassed about being wrong two out of every five tries. But Yale students, who do have reputations, searched for a hidden order in the sequence. They couldn’t deal with forty-per-cent error, so they ended up with almost fifty-per-cent error.

And if you read the article, it's instructive to consider whether you're a hedgehog or a fox.

John Dale : 13 Dec 2005 21:32 |  Tags: Musings |  Comments (2) |  Trackbacks (1) |  Report a problem