In the days before the 2016 US presidential election, nearly every national poll put Hillary Clinton ahead of Donald Trump—up by 3%, on average. FiveThirtyEights predictive statistical model—based on data from state and national voter polls—gave Clinton a 71.4% chance of victory. The New York Times model put the odds at 85%.
Trumps subsequent win shocked the nation. Pundits and pollsters wondered: How could the polls have been so wrong?
Trump-Clinton isnt the only example of a recent electoral surprise. Around the world, including in the 2015 United Kingdom election, the 2015 Brexit referendum, the 2015 Israeli election, and the 2019 Australian election, results have clashed with preelection polls.
But experts contend that these misses dont mean we should stop using or trusting polls. For example, postelection analyses of the 2016 US election suggest that national election polling was about as accurate as it has always been. (State polls, however, were a different story.) Clinton, after all, won the popular vote by 2%, not far from the 3% average that the polls found, and within the range of errors seen in previous elections. Polls failed to anticipate a Trump victory not because of any fundamental flaws, but because of unusual circumstances that magnified typically small errors.
“Everyone sort of walked away with the impression that polling was broken—that was not accurate,” says Courtney Kennedy, director of survey research at the Pew Research Center.
The issue may be one of expectations. Polls arent clairvoyant—especially if an election is close, which was the case in many of the recent surprises. Even with the most sophisticated polling techniques, errors are inevitable. Like any statistical measure, a poll contains nuances and uncertainties, which pundits and the public often overlook. Its hard to gauge the sentiment of an entire nation—and harder still to predict weeks or even days ahead how people will think and act on Election Day.
“As much as I think polls are valuable in society, theyre really not built to tell you whos going to be the winner of a close election,” Kennedy says. “Theyre simply not precise enough to do that.”
Would you like to take a survey?
Pollsters do their best to be accurate, with several survey methods at their disposal. These days, polling is in the midst of a transition. While phone, mail-in and even (rarely) door-to-door surveys are still done, more and more polls are happening online. Pollsters recruit respondents with online ads offering reward points, coupons, even cash. This type of polling is relatively cheap and easy. The problem, however, is that it doesnt sample a population in a random way. An example of whats called a non-probability approach or convenience sampling, the Internet survey panels include only people who are online and willing to click on survey ads (or who really love coupons). And that makes it hard to collect a sample that represents the whole.
“Its not that convenience Internet panels cant be accurate,” says David Dutwin, executive vice president and chief methodologist of SSRS, a research survey firm that has worked on polls for outlets such as CNN and CBS News. “Its just generally thought—and most of the research finds this—theres certainly a higher risk with non-probability Internet panels to get inaccurate results.”
With more traditional methods, pollsters can sample from every demographic by, for instance, calling telephone numbers at random, helping ensure that their results represent the broader population. Many, if not most, of the major polls rely on live telephone interviews. With caller ID and the growing scourge of marketing robocalls, many people no longer answer calls from unknown numbers. Although response rates to phone surveys have plummeted from 36% in 1997 to 6% in 2018—a worrisome trend for pollsters—phone polls still offer the “highest quality for a given price point,” Dutwin says.
In fact, most efforts to improve the accuracy of polling set their sights on relatively small tweaks: building better likelihood models, getting a deeper understanding of the electorate (so pollsters can better account for unrepresentative samples), and coming up with new statistical techniques to improve the results of online polls.
One promising new method is a hybrid approach. For most of its domestic polls, Kennedy says, the Pew Research Center now mails invitations to participate in online polls—thus combining the ease of Internet surveys with random sampling by postal address. So far, she says, its working well.
How polling goes wrong
In the spring of 2016, the American Association for Public Opinion Research formed a committee to look into the accuracy of early polling for the 2016 presidential race. After the election, the group turned to figuring out why the polls missed Trumps victory. One primary reason, it found, was that polls—particularly state polls—failed to account for voters with only a high school education. Normally, this wouldnt have mattered much, as voting preferences among people with different education levels tend to balance each other out. But in an election that was anything but normal, non-college-educated voters tilted toward Trump.
“College graduates are more likely to take surveys than people with less education,” says Kennedy, who chaired the AAPOR committee. “In 2016, peoples education levels were pretty correlated with how they voted for president, especially in key states.” This particular skew was unique to the 2016 election, but any kind of unrepresentative sample is the biggest source of error in polling, she says.
Pollsters have tools for predicting how skewed a sample might be and can try to correct for it by giving a proportionally larger weight to responses from any underrepresented groups. In the 2016 election, Kennedy says, many state polls didnt perform the proper statistical weighting to account for an underrepresentation of non-college-educated voters. Some pollsters, she says, might be reluctant to apply this kind of weighting, citing the fact that they dont know voters education levels ahead of time. Shoestring budgets also mean state polls tend to be of lower quality. (In 2016, their average error was 5.1%, compared to 2.2 for national polls.)
Even if pollsters can perfectly account for skewed samples, the responses to surveys could themselves be problematic. Questionnaires could be poorly designed, filled with leading questions. Or respondents might not be telling the truth. For example, voters may not want to admit their true voting preferences. This phenomenon is dubbed the Bradley Effect, named after the 1982 California gubernatorial election in which longtime Los Angeles Mayor Tom Bradley, an African American, lost to George Deukmejian, a white state official, despite leading in the polls.
According to the theory, voters would tell pollsters they would vote for the minority candidate to appear more open-minded, even though they would ultimately choose otherwise. Soon after the 2016 election, some election analysts suggested that a similar “shy Trump voter” effect may have been at play, although the AAPOR report didnt find much evidence that it was.
Telling the future
Its easy to write off the power of polls when they pick the wrong winner. But doing so misses the intended purpose (and acknowledged capability) of polling: to capture a snapshot of public opinion—not to make a prediction.
Most surveys ask people what they think about topics like education policy or a president's job performance at that moment in time. Election polls, on the other hand, ask people to forecast their future behavior. Asking voters whom they would pick if the election were held today—technically not about the future—poses a hypothetical situation nonetheless. This hypothetical nature of election polling is what makes it uniquely challenging.
“Were surveying a population that doesnt yet exist,” Dutwin says. “WRead More – Source
