The Signal and the Noise
Why So Many Predictions Fail-but Some Don't
"Nate Silver's The Signal and the Noise is The Soul of a New Machine for the 21st century."
—Rachel Maddow, author of Drift
Nate Silver built an innovative system for predicting baseball performance, predicted the 2008 election within a hair’s breadth, and became a national sensation as a blogger—all by the time he was thirty. The New York Times now publishes FiveThirtyEight.com, where Silver is one of the nation’s most influential political forecasters.
Drawing on his own groundbreaking work, Silver examines the world of prediction, investigating how we can distinguish a true signal from a universe of noisy data. Most predictions fail, often at great cost to society, because most of us have a poor understanding of probability and uncertainty. Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too. This is the “prediction paradox”: The more humility we have about our ability to make predictions, the more successful we can be in planning for the future.
In keeping with his own aim to seek truth from data, Silver visits the most successful forecasters in a range of areas, from hurricanes to baseball, from the poker table to the stock market, from Capitol Hill to the NBA. He explains and evaluates how these forecasters think and what bonds they share. What lies behind their success? Are they good—or just lucky? What patterns have they unraveled? And are their forecasts really right? He explores unanticipated commonalities and exposes unexpected juxtapositions. And sometimes, it is not so much how good a prediction is in an absolute sense that matters but how good it is relative to the competition. In other cases, prediction is still a very rudimentary—and dangerous—science.
Silver observes that the most accurate forecasters tend to have a superior command of probability, and they tend to be both humble and hardworking. They distinguish the predictable from the unpredictable, and they notice a thousand little details that lead them closer to the truth. Because of their appreciation of probability, they can distinguish the signal from the noise.
With everything from the health of the global economy to our ability to fight terrorism dependent on the quality of our predictions, Nate Silver’s insights are an essential read.
1. A Catastrophic Failure of Prediction
2. Are You Smarter Than a Television Pundit?
3. All I Care About is W's and L's
4. For Years You've Been Telling Us That Rain is Green
5. Desperately Seeking Signal
6. How to Drown in Three Feet of Water
7. Role Models
8. Less and Less and Less Wrong
9. Rage Against the Machines
10. The Poker Bubble
11. If You Can't Beat 'Em...
12. A Climate of Healthy Skepticism
13. What You Don't Know Can Hurt You
One of Wall Street Journal's Best Ten Works of Nonfiction in 2012
“Mr. Silver, just 34, is an expert at finding signal in noise… Lively prose — from energetic to outraged… illustrates his dos and don’ts through a series of interesting essays that examine how predictions are made in fields including chess, baseball, weather forecasting, earthquake analysis and politics… [the] chapter on global warming is one of the most objective and honest analyses I’ve seen… even the noise makes for a good read.”
—New York Times
“Not so different in spirit from the way public intellectuals like John Kenneth Galbraith once shaped discussions of economic policy and public figures like Walter Cronkite helped sway opinion on the Vietnam War…could turn out to be one of the more momentous books of the decade.”
—New York Times Book Review
"A serious treatise about the craft of prediction—without academic mathematics—cheerily aimed at lay readers. Silver's coverage is polymathic, ranging from poker and earthquakes to climate change and terrorism."
—New York Review of Books
"Mr. Silver's breezy style makes even the most difficult statistical material accessible. What is more, his arguments and examples are painstakingly researched..."
—Wall Street Journal
"Nate Silver is the Kurt Cobain of statistics... His ambitious new book, The Signal and the Noise, is a practical handbook and a philosophical manifesto in one, following the theme of prediction through a series of case studies ranging from hurricane tracking to professional poker to counterterrorism. It will be a supremely valuable resource for anyone who wants to make good guesses about the future, or who wants to assess the guesses made by others. In other words, everyone."
—The Boston Globe
"Silver delivers an improbably breezy read on what is essentially a primer on making predictions."
“The Signal and the Noise is many things — an introduction to the Bayesian theory of probability, a meditation on luck and character, a commentary on poker's insights into life — but it's most important function is its most basic and absolutely necessary one right now: a guide to detecting and avoiding bullshit dressed up as data…What is most refreshing… is its humility. Sometimes we have to deal with not knowing, and we need somebody to tell us that.”
“[An] entertaining popularization of a subject that scares many people off… Silver’s journey from consulting to baseball analytics to professional poker to political prognosticating is very much that of a restless and curious mind. And this, more than number-crunching, is where real forecasting prowess comes from.”
“Nate Silver serves as a sort of Zen master to American election-watchers… In the spirit of Nassim Nicholas Taleb’s widely read “The Black Swan”, Mr. Silver asserts that humans are overconfident in their predictive abilities, that they struggle to think in probabilistic terms and build models that do not allow for uncertainty.”
"Silver explores our attempts at forecasting stocks, storms, sports, and anything else not set in stone."
"The Signal and the Noise is essential reading in the era of Big Data that touches every business, every sports event, and every policymaker."
“Laser sharp. Surprisingly, statistics in Silver’s hands is not without some fun.”
“A substantial, wide-ranging, and potentially important gauntlet of probabilistic thinking based on actual data thrown at the feet of a culture determined to sweep away silly liberal notions like ‘facts.’”
—The Village Voice
“Silver shines a light on 600 years of human intelligence-gathering—from the advent of the printing press all the way through the Industrial Revolution and up to the current day—and he finds that it's been an inspiring climb. We've learned so much, and we still have so much left to learn.”
“Nate Silver’s The Signal and the Noise is The Soul of a New Machine for the 21st century (a century we thought we’d be a lot better at predicting than we actually are). Our political discourse is already better informed and more data-driven because of Nate’s influence. But here he shows us what he has always been able to see in the numbers—the heart and the ethical imperative of getting the quantitative questions right. A wonderful read—totally engrossing.”
—Rachel Maddow, author of Drift
“Yogi Berra was right: ‘forecasting is hard, especially about the future.’ In this important book, Nate Silver explains why the performance of experts varies from prescient to useless and why we must plan for the unexpected. Must reading for anyone who cares about what might happen next.”
—Richard Thaler, co-author of Nudge
“Making predictions in the era of ‘big data’ is not what you might imagine. Nate Silver's refreshing and original book provides unpredictably illuminating insights differentiating objective and subjective realities in forecasting our future. He reminds us that the human element is still essential in predicting advances in science, technology and even politics... if we were only wise enough to learn from our mistakes.”
—Governor Jon Huntsman
“Here's a prediction: after you read The Signal and the Noise, you'll have much more insight into why some models work well—and also why many don't. You'll learn to pay more attention to weather forecasts for the coming week—and none at all for weather forecasts beyond that. Nate Silver takes a complex, difficult subject and makes it fun, interesting, and relevant.”
—Peter Orszag, former director of the Office of Management and Budget
“Projection, prediction, assumption, trepidation, anticipation, expectation, estimation… we wouldn’t have 80 words like this in the English language if it wasn’t central to our lives. We tend not to take prediction seriously because, on some level, we know that we don’t know. Silver shows us how this inevitable part of life goes awry when projected on a grand scale into the murky worlds of politics, science and economics. Dancing through chess, sports, snowstorms, global warming and the McLaughlin Group, he makes a serious and systematic effort to show us how to clean the noise off the signal.”
—Bill James, author of The Bill James Baseball Abstracts
Can you explain the title of your book, THE SIGNAL AND THE NOISE?
It’s a metaphor that comes from electrical engineering. The signal is the sound that you want to transmit: say, a recording of Beethoven’s Moonlight Sonata. The noise is anything that interferes with it: say, the cracking from a nearby radio tower.
I found that this metaphor was coming up again and again in my research. Intelligence officials trying to detect terrorist activity will speak of a terrorist’s signal. Or an economist might speak of the noise in financial markets.
Their goal is to isolate the signal from the noise. But I call the book “The Signal and The Noise” because I found that there is a sort of duality between them. We can trick ourselves into thinking that random patterns are meaningful ones – that noise is signal – and sometimes vice versa.
Why do you think statistics books continue to capture the popular imagination, from Freakonomics to Moneyball?
We encounter so much information today that people are naturally curious about what in the heck we should do with all of it. And we’re becoming less trusting of institutions that mediate information, like the news media. We have all this data, and we want to learn for ourselves what it all means.
A little bit of math and statistics and probability and logic helps us with our information-processing goals. But what’s great about books like Moneyball and Freakonomics is that they make statistics approachable. Subjects like English and history are taught in very hands-on ways – you read great books, discuss the ideas and characters, and it’s easy to understand their relevance. Whereas math is taught in very abstract and technical ways – even though it’s just as relevant to our everyday lives, and just as intuitive if it’s taught well.
Books like Freakonomics and Moneyball help to bridge that gap. They’re sort of making up for the calculus teacher that had you memorize one too many derivatives and turned you off to the subject as a result. Not that there’s anything wrong with calculus.
Even before you came on the scene as a political forecaster, you developed an innovative system for predicting baseball performance – how did you go from sports to politics?
Partly because of Moneyball, the competition in baseball was getting very fierce. Most of the inefficiencies that Michael Lewis described in the book were exploited a long time ago. There’s hardly a team left that doesn’t employ a statistical analyst of some kind, or which doesn’t know that on-base percentage is more important than batting average.
Whereas in politics – well, you don’t need watch Fox News or MSNBC for very long to know that there’s a lot of hot air when it comes to political coverage. Your average political pundit is pretty detached from the things that normal voters care about. Reporters are a lot better than pundits, but they still need to weave messy information into neat narratives. So I thought there was room to apply a little bit more rigor to the topic.
What is the easiest professional sport to predict?
I’d argue that baseball is both the easiest and the hardest to predict, depending on how you’re thinking about the question. The statistical methods are the most complete in baseball. Pretty much everything that has happened on a baseball diamond within the past 150 years has been dutifully and accurately recorded. And for the most part, these statistics provide a very good description of what’s really going on in the game.
But we also know that there is a lot of luck in baseball. Even a world-class team will lose one third of its ballgames. And a .300 hitter will go through plenty slumps where he can’t hit a lick. So we know a lot about how to isolate the signal from the noise in baseball – but there is an awful lot of noise!
So maybe the best answer is something like tennis, which meets both definitions of predictability. It’s quite simple structurally, so you can describe it pretty well with statistics. And the best players – Roger Federer, Rafael Nadal – are absolutely dominant.
Politics and Baseball, the two subjects you are best known for, are just pat of the book. Why was it important to include so many different fields - economics, earth and life sciences, games, even terrorism?
One thing that baseball fans know is to be wary of small sample sizes. If you show up at the ballpark, and the catcher gets three hits that day, that doesn’t really tell you very much about how good he really is. It takes a long time – hundreds of at-bats – for the signal to emerge since there’s so much luck in the game.
But in the same way, I thought, perhaps baseball is an exceptional case. Are there Moneyball-like success stories in other fields in which statistics and analysis and prediction is pertinent?
In fact, I found that there are entire disciplines in which our analysis has failed to produce much progress, at least as measured by our ability to make reliable predictions. Finance and economics are obvious examples of this, for instance. Economists have literally tens of thousands of data series to mine – more statistics than baseball geeks do. But they still aren’t able to predict recessions more than a few months in advance.
The book needed to cover a diverse enough range of examples that I could to some systematic conclusions about why predictions succeed and why they fail in different fields. And that meant going beyond the cases that were most familiar to me when I started to write and research the book.
Often, the conclusions were surprising. I’d thought that weather forecasting was a hopeless case, for instance, but it turned out to be a huge success story. Meteorologists and professional gamblers basically emerge as the heroes of the book.
Rather controversially you say in the book “we can never make perfectly objective predictions. They will always be tainted by our subjective point of view.” How so?
Well, I think human beings are pretty darned smart. Our brains can store about three terabytes of information, which is just an enormous amount. And a three-year old is able to do a lot of things that a supercomputer can’t.
Still, three terabytes represents only about one one-millionth of the information that IBM says is now being produced in the world each day. We have to be terribly selective about the information that we choose to remember. That necessarily implies that we have a point of view – the set of facts that I have at my command won’t be the same as yours. And we have to make approximations, whether it’s in the form of our language, or the mathematical models that we design.
Even the things we take most for granted, like our sensory inputs (vision, hearing, etc.) rely heavily on making approximations about the objective world. We’re just taking in way more information out there than our brains can process.
So it’s absolutely delusional to think that any one of us has a monopoly on the truth – that our beliefs about the world aren’t flawed in any number of large and small ways. The book, in some ways, is about accepting our flaws, as well as recognizing the things that we’re good at.
What distinguishes those who are good at forecasting to those who seem to get it all wrong?
The whole book is an answer to that question. But here’s one big thing that weather forecasters and gamblers – two of our success stories – have in common. They recognize that their knowledge of the world is imperfect. They express their predictions in terms of probabilities: there’s a 40 percent chance of rain tomorrow; there’s a 30 percent chance that I’ll catch a card to make a flush and win a huge poker hand.
This type of thinking turns out to be extremely important when it comes to sorting through the enormous amount of information that we encounter today.
What do you feel most accounted for the grand failure of forecasting the 2008 financial crisis?
A lot of things had to go wrong to create such a gigantic mess. The credit-rating agencies built models that assumed that the status of one’s person mortgage wasn’t much related to another’s: if a carpenter in Cleveland defaults on his mortgage, that has no effect on whether a dentist in Denver does. That assumption fails miserably when you have a massive housing bubble, and mortgages to go underwater all across the country.
What’s worse, people doubled-down rather than hedge their bets. For every dollar invested in purchasing a home, Wall Street was making about $50 in bets on the side. So a collapse in housing prices brought down the entire financial system. But it began with having a naïve trust in models that made exact-seeming predictions based on utterly flawed assumptions.
In 2008 you correctly predicted Barak Obama’s victory in 49 of 50 states as well as the winner of all 35 U.S. Senate races. In the four years since, has your methodology changed at all when it comes to politics? What’s toughest about political forecasting?
What’s tricky about presidential elections is simply that they don’t happen very often – just one every four years. And there are dozens of factors that go into determining the winner. It’s very much the opposite of something like baseball, where each team plays 162 games per season, but the structure of the game is pretty simple. In presidential elections, we’re really just making educated guesses about which factors determine the winners and losers.
With that said, my methodology is increasingly starting to incorporate some of those structural factors in addition to polling. The evidence is pretty clear that a bad economy, for instance, makes life challenging for the incumbent. If there’s a bad jobs report, but a good poll for Obama in Ohio, the jobs report is often the more important data point.
Let me protest, incidentally, that some of those forecasts were a bit fortunate in 2008. We had Al Franken with a 50.1 percent chance of winning the Senate race in Minnesota – basically a coin flip. The coin came up the right way for me after a prolonged recount there. But that’s basically just luck, so it’s no reason to rest on one’s laurels.
In the era of Big Data and the advent of faster, better technology, are we getting better at forecasting?
In certain circumstances, yes, the technology is helping – like in baseball prediction or weather forecasting. But it shouldn’t be assumed that this is the default case.
Think of all the disasters that we’ve had in the new millennium – from the financial crisis to the September 11 attacks to the Japan earthquake (some seismologists thought we couldn’t possibly have so large an earthquake there). All of these involved some substantial failure of prediction, and all of these occurred in information-rich fields.
More information and better technology aren’t all that useful if we don’t know how to use it. And all that shiny the new technology can sometimes make us overconfident – we think we have mastery of a subject when we just don’t.
How is forecasting a natural phenomenon (weather, earthquakes) different from predicting human performance (financial markets, sports, politics)?
Many natural systems, including weather and earthquakes, are quite complex. But at least they aren’t changing much, at least on the scale of human lifespans. Slowly, if not always steadily, we’ve made progress in predicting natural phenomenon.
When it comes to systems that involve how humans interact with one another, however, they’re not just complex – but also growing more complex all the time. There’s just no comparison between the globalized economy of today and the localized, agrarian economy of a couple of centuries ago. Seven billion human beings, who can catch a flight to anywhere in the world if they have the means, can spread the flu around the planet much faster they did a generation ago.
So we’re always running against a moving target. Prediction in these areas requires a somewhat more defensive posture: we have to prepare more for “unknown unknowns” as human beings keep finding new and ingenuous ways of interacting with one another.
You write about hedgehogs and foxes. Can you explain the difference? Which one are you? Obama? Romney?
These terms come from a quote attributed to the Greek poet Archilochus: "the fox knows many things, but the hedgehog knows one big thing". Foxes tend to be multidisciplinary and adaptable, always looking for little gains around the margin. They’re comfortable with probability and uncertainty. They’re a little more pragmatic.
Hedgehogs like to swing for the fences and seek out the big kill. They want to find some grand unifying principle that explains the world. But they can be stubborn and ideological as a result.
I’m sure that we need both these types of people in the world. But in the book, I cite research to suggest that the foxes are a lot better at making predictions. They’re more likely to know their limits, and less likely to mistake noise for a signal because it happens to fit some theory they’ve concocted.
Obama and Romney are both foxes, I think. I’m sure that some partisans will disagree, but I think they’re both quite pragmatic – perhaps unusually so for presidential candidates. Certainly as compared to George W. Bush. I’m a fox too.
To keep up-to-date, input your email address, and we will contact you on publication
Please alert me via email when: