Predicting the Future
How to do better on this year's forecast. Quiz A.
I was going to send this out yesterday, but as I was working on it, I began to whether there was any obvious way we could do better this year. We didn’t do terribly last year, but I think we under-performed the prediction markets. (Unfortunately, I didn’t make a record of my own predictions—and I genuinely don’t remember what they were—so I don’t know if I did better or worse than you did.)
My readers should be able to do as well, if not better, as the prediction markets. If you can’t, then I’m doing something wrong: It’s my job to keep you well-informed. That’s why you subscribe.
But as a team led by Barbara Meller noted in a paper titled “Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions,” people are, in general, very bad at predicting in the future:
… [T]he psychological literature, in which the accuracy of subjective probability judgments has been examined, raises serious doubts about people’s competence as intuitive forecasters. People often misuse Bayes’s Theorem when updating their beliefs (Birnbaum & Mellers, 1983; Kahneman & Tversky, 1985). They test hypotheses in suboptimal ways (Ofir, 1988; Wason, 1968). They are susceptible to the hindsight bias or the “I-knew-it-all- along” effect (Christensen-Szalanski & Wilhelm, 1991). Furthermore, people are often overconfident about what they know, consistent with self-serving biases and positive illusions (Baron, 2000; Gilovich, Griffin, & Kahneman, 2002; Lichtenstein, Fischhoff, & Phillips, 1982). Entrepreneurs believe their personal chances of creating a successful business are much higher than statistics suggest (Cooper, Woo, & Dunkelberg, 1988), and automobile drivers believe they are safer and more skilled behind the wheel than their peers (Svenson, 1981).1 Even experts, such as skilled basketball players, are overly optimistic about their chances of future successes (Jagacinski, Isaac, & Burke, 1977).
There are however, ways to improve the situation. In 2011, the US Intelligence Community sponsored a series of geopolitical forecasting tournaments. Participants were asked questions much like the ones on our 2024 quiz. They discovered that some people were able to predict political events both more accurately and further into the future than others—in fact, these people were 30 percent more accurate than trained intelligence analysts, despite lacking their access to classified information. They called these people superforecasters. The term was then popularized by political psychologist Philip Tetlock in his book Superforecasting: The Art and Science of Prediction.
Intrigued by this finding, the aforementioned researchers conducted a multi-year study of superforecasters, creaming off the top talent and assigning them to elite teams. “Defying expectations of regression toward the mean two years in a row,” they reported, “superforecasters maintained high accuracy across hundreds of questions and a wide array of topics.”
What, they wondered, made them good at this? You will not be surprised to learn that the superforecasters were highly intelligent. Fluid intelligence was the strongest predictor of accuracy: They all scored at least a standard deviation higher than the general population. They also scored at least a standard deviation higher on crystallized intelligence, and even higher on tests of political knowledge.
But a particular cognitive style, they found, also lent itself to skill in forecasting. A battery of psychological tests indicated that the superforecasters were unusually competitive. They took unusual pleasure in solving problems, and they were unusually open-minded. They were inclined toward “a scientific worldview with little tolerance for supernaturalism.” They were less prone to irrational biases, like the anchoring effect. Their forecasts were more precise: whereas control subjects, when asked to estimate the probability of an event, tended to choose multiples of 10, the super forecasters favored numbers like 73.
Tetlock’s research—particularly through the Good Judgment Project, which began as part of the intelligence community’s tournament—suggests practical advice for improving forecasting skills. These are the principles he advises:
Express predictions in terms of probabilities rather than binary outcomes. For example, instead of saying, “Netanyahu will no longer be the Prime Minister,” assign a percentage likelihood. When uncertain, provide a range of probabilities. Regularly compare your forecasts to real outcomes to calibrate yourself, particularly to judge whether you’re overconfident (or under-confident).
Break complex questions into simpler, more manageabl components.Instead of asking, “Will the United States experience a civil war,” ask:
What are its social and economic conditions?
How often have similar conditions led to civil wars historically?
What warning signs may currently be observed?
Use Fermi Estimation. Make rough, order-of-magnitude estimates by breaking a question into smaller factors you can estimate.
Start with base rates, then adjust for specifics. Anchor on historical data: Begin with statistical averages or historical frequencies for similar events. If predicting the likelihood of a coup in Country X, look at the global rate of coups in similar circumstances as a starting point. Once you have a base rate, refine it using the details unique to the Country X.
When new evidence arises, revise your probabilities accordingly instead of clinging to your initial estimate.
Avoid overreacting to single data points; assess the quality and context of the information.
Actively look for evidence and opinions that challenge your assumptions.
Collaborate with people who offer different expertise and viewpoints. (But avoid groupthink by ensuring diverse perspectives are heard.)
Be modest about certainty. Know the limits of your knowledge and avoid absolute predictions.
Counteract anchoring. People are prone to allowing an initial piece of information overly to influence their estimates. Don’t do that.
Overcome hindsight bias, too. Recognize that outcomes often seem obvious in retrospect but were not necessarily predictable beforehand.
Record your forecasts, the reasoning behind them, and the probabilities you assign. Review your track record regularly to identify patterns of success and failure.
Study your correct and incorrect predictions to refine your approach.
Ask others to identify blind spots to improve your calibration.
Use structured prediction markets to test and refine your skills.
Use existing statistical or econometric models as inputs.
Explore multiple possible futures rather than fixating on a single outcome. Ask, “What would have to be true for this prediction to succeed or fail?”
Specify the conditions under which your probability would increase or decrease.
None of it is really all that surprising, but few of us actually do it.
Skill in forecasting, therefore, is not just a matter of being well informed, but practicing certain cognitive habits. Without more information about why you made the predictions you did, it’s hard for me to diagnose what went wrong, precisely. Was it lack of information? Or cognitive error?
But here are a few observations. Some of my questions were just badly worded. For example, when I asked whether the Israel-Hamas war would trigger a full-blown regional conflict, I should have defined “a full-blown regional conflict.” It triggered a regional conflict, for sure. Iran lobbed hundreds of missiles and drones at Israel (intercepted by the US, Jordan, and Saudi Arabia); Israel assassinated Iranian generals in Syria and Lebanon and Hamas leaders in Iran; it decapitated Hezbollah, invaded Lebanon, took out of Iran’s air defenses, struck the Houthis in Yemen, and set off of a chain of events that not only caused the Assad regime to collapse, but sent Russia and Iran hightailing it out of Syria. (A fine year’s work.) But what does “full-blown” mean? Does it mean “total war, with no restraints?” “Nuclear war?” Does it mean surpassing a certain casualty figure? I suppose either answer could be correct. My fault. I phrased the question poorly.
But why did 80 percent of you think that Netanyahu would be unseated by the end of the year? Netanyahu’s ability to cling to power is notoriously one of the world’s great natural wonders. Without knowing why you thought he was a goner, it’s hard for me to guess what kind of error was involved. Wishful thinking? A sense that after presiding over October 7, he should be evicted? (In any event, you clearly didn’t begin with the base rate.)
Normalcy bias and wishful thinking, I suspect, account for the most striking errors. For example, only 6 percent of you correctly predicted that Trump, but not Biden, would be on the ballot. Only 32 percent of you correctly predicted that Trump would be the next president. (I can’t remember what I thought, but I bet I wasn’t one of them. I really didn’t think it would happen. I didn’t want to think it would happen.)
There were a few more responses I just didn’t understand. Why did 40 percent of you predict that 2024 would not surpass 2023 as the hottest year on record? “The global average temperature will go up this year” is always the safe bet. That question was supposed to be a giveaway. Seeing as only 3 percent of you predicted a nuclear exchange, what was your theory about why temperatures would go down?
For a number of questions, I think we deserve partial credit, because the events in question—North Korea’s seventh nuclear test, the release of ChatGPT 5, renewables overtaking coal in global energy production—are expected any day now. We wouldn’t get partial credit in the betting markets, but we’re not a casino: There’s no reason the house always has to win.
I can’t, unfortunately, create a quiz that allows you to express your predictions in terms of probability or break your answers into smaller components. (The only option Substack gives me is “multiple choice.”) So again, this year, we have yes/no questions, but with this clarification: If you say that an event will happen, it means, “The odds of it happening are better than 50 percent.”
But let’s do things a little differently this year.
I’ll be sending you the quiz twice: once in this newsletter, and once in a newsletter you’ll get about five minutes later. They’ll be exactly identical, but we we’ll call them Quiz A and Quiz B.
Take Quiz A, in this newsletter, right now. Just wing it. Don’t overthink it. Quiz A will be open for 24 hours.
Don’t respond to the Quiz B right away. I’m going to leave it open for a full week. Give this one more serious thought. Use the comments to work out your answers, using all of the techniques above.
For Quiz B, you’re not only allowed to collaborate with each other in the comments, you’re strongly encouraged to do so. Try doing everything Tetlock suggests, including breaking the problem down to its more manageable components, starting with the base rate, availing yourself of publicly available statistical and econometric data, searching for evidence that might contradict your view, and discussing your predictions with people who don’t agree with you.2 Explain your thought process in some detail. Once you’ve done this, take Quiz B. I’ll do the same, but I won’t discuss my predictions with you until you've taken Quiz B. I don’t want to bias you. I’ll discuss them afterwards.
Or just take Quiz A, if you feel like taking Quiz B wouldn’t be fun. (The research suggests that if this isn’t the kind of thing you like to do, odds are you’re not good at it. So don’t do it out of a sense of obligation: You’ll just bring our score down.)
I’ll be curious to know if our answers to Quiz A and Quiz B are significantly different. And I’ll be even more curious to dig this out next year to see how our predictions compared to reality—and whether we beat the prediction markets.
Quiz A
(Take this one now.)
Stay tuned for Quiz B.
An astonishing 93 percent of Americans believe they’re better than average drivers. (Not me. I’d guess I’m in the bottom 10 percent, spared an even lower ranking only because I try to be sober when I drive.)
Don’t forget: choose your parents carefully, given the importance of fluid intelligence to this task.
As of today, most questions are still open for 3 more days, but quite a few of them are already closed?
Interesting poll, I was surprised to see agreement with other respondents on many of the questions.