some fresh ideas For my understanding of the Good Judgment Project, I am indebted to Barbara Mellers et al., “Psychological Strategies for Winning a Geopolitical Forecasting Tournament,” Psychological Science 25, no. 5 (2014): 1106–15; Daniel Kahneman, “How to Win at Forecasting: A Conversation with Philip Tetlock,” Edge, December 6, 2012, https://edge.org/conversation/how-to-win-at-forecasting; Michael D. Lee, Mark Steyvers, and Brent Miller, “A Cognitive Model for Aggregating People’s Rankings,” PloS One 9, no. 5 (2014); Lyle Ungar et al., “The Good Judgment Project: A Large Scale Test” (2012); Philip Tetlock, Expert Political Judgment: How Good Is It? How Can We Know? (Princeton, N.J.: Princeton University Press, 2005); Jonathan Baron et al., “Two Reasons to Make Aggregated Probability Forecasts More Extreme,” Decision Analysis 11, no. 2 (2014): 133–45; Philip E. Tetlock et al., “Forecasting Tournaments Tools for Increasing Transparency and Improving the Quality of Debate,” Current Directions in Psychological Science 23, no. 4 (2014): 290–95; David Ignatius, “More Chatter than Needed,” The Washington Post, November 1, 2013; Alex Madrigal, “How to Get Better at Predicting the Future,” The Atlantic, December 11, 2012; Warnaar et al., “Aggregative Contingent Estimation System”; Uriel Haran, Ilana Ritov, and Barbara A. Mellers, “The Role of Actively Open-Minded Thinking in Information Acquisition, Accuracy, and Calibration,” Judgment and Decision Making 8, no. 3 (2013): 188–201; David Brooks, “Forecasting Fox,” The New York Times, March 21, 2013; Philip Tetlock and Dan Gardner, Seeing Further (New York: Random House, 2015).
A group of At various points during the GJP, the precise number of researchers involved fluctuated.
questions as the experts In response to a fact-checking email, Barbara Mellers and Philip Tetlock, another of the GJP leaders, wrote: “We had two different types of training in the first year of the tournament. One was probabilistic reasoning and the other was scenario training. Probabilistic reasoning worked somewhat better, so in subsequent years, we implemented only the probabilistic training. Training was revised each year. As it evolved, there was a section on geopolitical reasoning and another on probabilistic reasoning….Here is a section that describes the training: We constructed educational modules on probabilistic-reasoning training and scenario training that drew on state-of-the-art recommendations. Scenario training taught forecasters to generate new futures, actively entertain more possibilities, use decision trees, and avoid biases such as over-predicting change, creating incoherent scenarios, or assigning probabilities to mutually exclusive and exhaustive outcomes that exceed 1.0. Probability training guided forecasters to consider reference classes, average multiple estimates from existing models, polls, and expert panels, extrapolate over time when variables were continuous, and avoid judgmental traps such as overconfidence, the confirmation bias, and base-rate neglect. Each training module was interactive with questions and answers to check participant understanding.”
abilities to forecast the future In response to a fact-checking email, Don Moore wrote: “On average, those with training did better. But not everyone who got trained did better than all the people who did not get it.”
“tremendously useful” Brooks, “Forecasting Fox.”
“things you aren’t sure about” In response to a fact-checking email, Don Moore wrote: “What makes our forecasters good is not just their high level of accuracy, but their well-calibrated humility. They are no more confident than they deserve to be. It’s ideal to know when you have forecast the future with accuracy and when you haven’t.”
or roughly 20 percent In an email, Howard Lederer, a two-time World Series of Poker champion, explained the further nuances required in analyzing this hand: “The hand you use as an example is MUCH more complicated than it appears.” Given what’s known, Lederer said, there is actually a better than 20 percent chance of winning. “Here’s why. If you KNOW your opponent has an A or a K, then you know seven cards. Your two [cards], your opponent’s one card, and the four [communal cards] on the board. This means there are 45 unknown cards (you have no information on your opponent’s other card). This would mean you have nine hearts to win, and 36 non-hearts to lose. The odds would be 4 to 1, or 1 in 5. The percentages are 20%. As long as you are not putting more than 20% of the money into the pot, it’s a good call. Here’s where you might ask: if I am only 20% to win against an A or K, then how can I be better than [20%] to win? Your opponent might not have an A or K! He could have a spade flush draw without an A or K, he could have a straight draw with a 5–6. He could have a lower heart draw. That would be great for you! There’s also a chance he just has garbage and is trying to bluff you with nothing. In general, I’d calculate the chances that your opponent has one of these drawing or bluffing hands at about 30% (given how many of these possibilities there are). So let’s do some probabilistic math: 70% of the time he has an A or K, and you win 20% of those times. 25% of the time he has a draw and you win about 82% of those hands (I’m combining various possible odds given his range of holdings when he is drawing). And 5% of the time he has a total bluff and you win 89% of the time when he has garbage. Your total chances of winning are: (.7 × .2) + (.25 × .82) + (.05 × .89) = 39%! This is a simple ‘expected value’ calculation. You can see that the .7, .25 and .05 part of the calculation adds up to 1. Meaning we have covered all the possible holdings and assigned them probabilities. And we are making our best guess as to our chances against each holding. At the table, you don’t have time to do all the math, but ‘in your gut’ you can feel the odds and make the easy call. One other note, if you miss your flush and your opponent bets, you should seriously consider calling anyway. You will be getting well over 10–1, and the chances he is bluffing are probably higher than that. This is just a simple taste of the complexity of poker.”
they’ll quit For more on calculating odds in poker, please see Pat Dittmar, Practical Poker Math: Basic Odds and Probabilities for Hold’em and Omaha (Toronto: ECW Press, 2008); “Poker Odds for Dummies,” CardsChat, https://www.cardschat.com/odds-for-dummies.php; Kyle Siler, “Social and Psychological Challenges of Poker,” Journal of Gambling Studies 26, no. 3 (2010): 401–20.
“odds work for you” In response to a fact-checking email, Howard Lederer wrote: “It’s more complex than that. Amateurs players make many different kinds of errors. Some play too loose. They crave the uncertainty and favor action over prudence. Some players are too conservative, favoring a small loss in a hand over taking the chance to win, but also the chance to take a large loss. Your job as a poker pro is to simply play your best each hand. In the long run, your superior decisions will defeat your opponent’s poor decisions, whatever they may be. The societal value of poker is that it is a great training ground for learning sound decision-making under conditions of uncertainty. Once you get the hang of playing poker, you develop the skills necessary to make probabilistic decisions in life.”
Annie’s brother, Howard Though it does not bear on the events described in this chapter, disclosure compels mentioning that Lederer was a founder and board member of Tiltware, LLC, the company behind Full Tilt Poker, a popular website that was accused of bank fraud and illegal gambling by the U.S. Department of Justice. In 2012, Lederer settled a civil lawsuit with the Department of Justice related to Full Tilt Poker. He admitted no wrongdoing, but did agree to forfeit more than $2.5 million.
winning this hand Technically, Howard has an 81.5 percent chance of winning—however, because it is hard to win half a hand of poker, this has been rounded up to 82 percent.
remaining cards on the table In response to a fact-checking email, Howard Lederer wrote: “I would say that in a 3 handed situation, [a pair of sevens] is close to 90% to be best before the flop. This is the hand where I agree anyone would have played her hand and my hand the same way; all in before the flop. After we had all the money in, I am not a slight favorite, but instead a large favorite. This [is] a unique feature of hold’em. If you have a slightly better hand than your opponent, you are often a big favorite. 7–7 is about 81% to beat 6–6.”
“they tell you might occur” In response to a fact-checking email, Howard Lederer wrote: “It’s not an easy thing to choose a profession where you lose more often than you win. One has to focus on the long run, and realize that if you get offered 10–1, on enough 5–1 shots, you will come out ahead, while also realizing that you will lose 5 out of 6 times.”
humans process information Tenenbaum, in an email responding to fact-checking questions, described his research this way: “Often we start with what looks like a gap between humans and computers, where humans are outperforming standard computers with intuitions that may not look like computations….But then we try to close that gap, by understanding how human intuitions actually have a subtle computational basis, which then can be engineered in a machine, to make the machine smarter in more human-like ways.”
“seeing just a few examples” Joshua B. Tenenbaum et al., “How to Grow a Mind: Statistics, Structure, and Abstraction,” Science 331, no. 6022 (2011): 1279–85.