Member Newsletter
This is a post not so much focused on the news of the moment but one in response to a question I get a lot. It’s also a post I’ve wanted to do because I’ll be able to refer back to it as we go forward through the final sprint of the campaign. The question is a really basic one: Given what happened in 2016 and 2020, how much confidence can we have that the current polls are giving us an accurate or realistic picture of the current campaign?
Let me deal briefly with what are important but mostly obvious caveats. Polls, or really poll averages, are almost never exactly right and not infrequently they suffer from systemic error. So can we rely on them? No. That would be silly. Most of the time they are fairly accurate predictors of election outcomes. But in close races, a “normal” polling miss of a point or two can change the result. But what people who ask me this question are really asking is whether we should expect that polls are underestimating Trump’s strength as they did in 2016 and 2020.
My tentative answer to this is, no, I don’t think we should. But how I get to that answer takes some explaining. And before the explaining we need some history.
Most of us remember that the polls missed Donald Trump’s shock victory in 2016 and then, in a near heartbreak, a similar miss happened again in 2020. But this understanding is not exactly right. And the way many people’s memories differ from what really happened is worth reviewing.
The polls really weren’t that far off in 2016. There were several sites doing poll averages that year (including TPM — the dear, departed TPMPolltracker). There are also different datasets — ones for two-way, three-way and four-way polls. So there’s no one average. But the great majority predicted that Clinton would win with a bit over 3 percent of the vote. She won the popular vote by 2.1 percentage points. That’s a miss of course and pretty much everyone was shocked that Trump won. But the result being one percentage point or maybe 1.5 percentage points off is about as close as you can hope for. It’s well within normal variability. For context, the final RCP average in 2012 had Barack Obama up by .7 percentage points. But the final result had Obama winning by 3.9 percentage points. The key was that in 2016 Trump won a series of critical states by very narrow margin.
So 2016 was the big shock and heartbreaker. But the polling miss was actually substantially larger in 2020, despite the fact that the predicted winner, Joe Biden, won. 538’s final average had Biden up by 8.4 percentage points; RCP had him up by 7.2 percentage points. But the actual result had him up only 4.5 percentage points. In other words, a substantially bigger miss than 2016. But not a wildly bigger miss than in 2012. That one just went in the other direction.
Given that two successive elections underestimated Trump why shouldn’t we expect it again?
A few reasons.
The first is that after the 2020 election, most pollsters went back into their data and made changes to their methodologies to slightly amplify the voices of those Trump-supporting demographics — generally, white non-college voters, with low social trust — which polls were missing. The consensus of most pollsters, from what I can tell, is that they’re pretty confident they’ve removed this potential source of systemic error. Obviously they may be wrong. But I suspect they’re not. I have a slightly different theory, for which there is a fair amount of evidence. And that is that the Dobbs decision overturning Roe v Wade was an inflection point. Before that, polls were generally underestimating Republican strength and since then they’ve more frequently been underestimating Democratic strength. We’ve seen this happen again and again.
A fair number of poll analysts will tell you that despite the predictions of a red wave, the polls actually got 2022 right. This is basically wrong, or at least involves a great deal of after-the-fact excuse making and special pleading to get the numbers into alignment. What happened in 2022 was that a lot of historical fundamentals predicted a Democratic bloodbath — president’s first midterm, unpopular president, iffy economy, etc. And for much of the year polls validated those fundamentals. Very late in the campaign the so-called “congressional generic ballot” did move back in the direction of a much closer race. That’s the basis of pollsters saying the polls got it right. But congressional generic is a fuzzy measure. And polls missed quite a few big Senate and gubernatorial races that Democrats won.
The response to my theory is that it’s not really a post-Dobbs switch. It’s that Democrats now have the more reliable voters. So they do better in special elections and off-year elections. And that doesn’t apply in a high-turnout presidential election. That may apply in low-turnout special elections. But it’s not a very good explanation of why polls underestimated Democratic strength in 2022 and 2023. Others say, well, it’s something that only applies when Trump is literally on the ballot. But again, that doesn’t really bear out. And to me it’s more a theory meant to retrofit the limited data to the idea that Trump will always have unpredictable strength. Another potential explanation is that it’s not or not only a post-Dobbs inflection: the rejiggering of methodologies to capture some “missing” Trump voters may have overshot the mark and started overestimating Trump or MAGA Republican strength. My hunch is that it’s probably a bit of both.
In any case, these are the reasons I don’t think we should expect or be overly worried that polls will underestimate Trump’s strength. That doesn’t mean it won’t happen. I’m just saying I don’t think it’s any more likely, based on the evidence before us, than that they’ll underestimate Harris. On election night in November I will definitely be worried that it will happen. But that’s because I worry about things and the stakes are super high. That’s not the same as thinking it’s likely to happen when I look at the available evidence.
At the end of the day, we don’t know what will happen. Polling is an imperfect craft, prone to errors of various sorts. An even bigger issue is one that impacts every effort to find patterns in recent elections. We simply don’t have remotely big enough data sets. People who look at Trump having some persistent ability beat the polls are talking about a grand total of two elections. In one of those elections, the polls actually weren’t off by that much. Meanwhile, 2020 was the first American election in modern history carried out during a still raging pandemic. It’s always seemed to me — and it’s not clear why more people don’t think this — that there’s good reason to see the entire 2020 general election as sui generis because of the way the pandemic impacted turnout and pre-election voter canvassing. But the mix of methodology changes pollsters made post-2020, which were substantial, combined with the results since then make me think that if anything we are in a period where polls tend to underestimate Democratic strength.