Science Fact: Prediction promises us the big answers: Will the stock market hit 35,000? Who will win the World Series? Will the Russians invade Crimea? (Oh, wait, that already happened.)
Prediction also offers us practical advice: You can’t tune into media without encountering a prediction. That your wrinkles or depression or arthritis will be improved by a brand name product. That your savings can be coaxed to grow into an idyllic retirement. That passing, or failing to pass, some piece of legislation will create an unprecedented economic or human or moral disaster.
We rightfully receive these claims with skepticism, yet they are appealing because we long for certainty. We really want to know what the future holds and how we can make it better. And we want that so much that we are ready to believe folks who claim to know the answers.
Prediction involves the future and science doesn’t really know what the future holds. Yet prediction can be studied and important lessons can be drawn. Surprisingly, one of the things science tells us is how to tell in advance whose crystal ball is most accurate.
Sixty years ago a Russian-British philosopher, Isaiah Berlin, wrote an essay that defined two different styles of looking at the world. He started by quoting the Greek poet Archilochus: “The fox knows many things, but the hedgehog knows one big thing.” A straightforward interpretation of these words is that the fox is clever, yet all his slyness may be defeated if the hedgehog rolls himself into a spiny ball to avoid becoming dinner.
Instead of focusing on predator and prey, Berlin uses the animals to represent two ways of seeing things. Some people see the world in terms of a single coherent vision, a system if you like, that guides their beliefs and their actions. These are the “hedgehogs,” people who instinctively feel that there are fundamental principles that determine what happens in the world. In the realm of politics, a hedgehog might have a passionate commitment to conservatism or progressivism. In economics, he might believe in the overriding power of the free market or of government planning. In social issues, a hedgehog might feel that religion or morality works as an irresistible force.
Conversely, there are the “foxes,” whose who “know many things.” The foxes bring no preconceived world view to the table: they gladly take in data and ideas from every available direction, whether they sympathize with the source or not. They have principles and beliefs, surely, but those are shaded, colored by circumstances and capable of being re-framed in various ways. Foxes can even find acceptable ways to address forbidden topics such as the monetary value of human life.
Why do we need this additional way to sort out people whom we probably haven’t even met yet? Because it tells us something about their ability when it comes to prediction.
Philip Tetlock, a psychology and management professor at the University of Pennsylvania, ran a twenty-year study to test how well experts can predict events in international affairs. He collected forecasts made by 284 world experts, 28,000 in all, and kept track to see which ones came true. The forecasts covered both large and small events occurring within or between countries, both long term and near term.
What Tetlock found was that “hedgehogs” – those with a strong political position, either liberal or conservative – were willing to make more radical predictions. The breakup of Yugoslavia. Russian invasion of the Baltic states. And sometimes they were spectacularly correct. However, they were wrong an overwhelming percentage of the time. The “foxes” – those who tended to see many sides of every issue – generally did not try predict such tornado-like events. However, their batting average was much better at the predictions they did make.
Analogous results were found in a study by Denrell and Fang of the University of Oxford and New York University. Instead of studying prediction in international politics, they looked at business decisions: Will a new product be a hit? How will the US economy perform during the coming year? They ran laboratory experiments to study prediction styles and confirmed their results by studying seven issues of the semiannual Wall Street Journal Survey of Economic Forecasts. They found that if you look at all the predictions people make, accurate prediction indicates good skill at forecasting. However, if you only look at extreme predictions, then accurate prediction actually indicates that the person has poor forecasting ability and just happened to get lucky.
The most telling way to summarize what all the researchers found is given in a description of a video talk by Tetlock:
The political expert who bores you with a cloud of “howevers” is probably right about what’s going to happen. The charismatic expert who exudes confidence and has a great story to tell is probably wrong.
Science Speculation: What are we to make of those spectacular predictions that came true? A one-page essay by James Surowiecki in The New Yorker lists a number of stock market predictions that were dramatically correct, but whose authors badly missed the boat the next time they opened their mouths. To explain that, consider the risks and the rewards. If I make a dramatic prediction and it comes true, I may be able to coast for some time on the coattails of my fame. But if it doesn’t come true, there are always excuses that can be given, and soon all is forgotten.
In our 7/24 culture of constant media there’s no punishment for being wrong but there’s a big payoff for being right. A rational person who makes his living by being in the public eye might decide that it’s worth stretching himself to make an extreme prediction. It’s exciting, it’s newsworthy and best of all, there’s hardly any downside.
In this environment, it’s amazing if we can find anyone at all who takes a careful look at all the factors and gives a nuanced answer. Think back to the most recent politician whom you heard give a thoughtful non-black-and-white reading on a difficult issue (Jon Huntsman might be an example). Then think about how quickly that person was drowned out by a chorus of partisan voices giving simple solutions that could fit into a sound bite.
A scientist is more likely to be a “fox,” offering a carefully qualified answer to a complex question. The media and the public want a “hedgehog” with a simple, direct answer. Perhaps it’s no wonder that it’s a rare scientist who can speak out publicly with grace and success.
Does anyone do a good job of prediction today? How often have you heard a prediction success that the same person followed with another success?
Drawing Credit: adapted from BenBois, on openclipart.org