In my last post I mentioned this data analysis GOP pollster and consultant Pat Ruffini has put together looking at Senate polling going back through the last four cycles, so 2018 through 2024. Ruffini is a very partisan Republican in all the meanings of that word. But I think of him as broadly reliable when it comes to compiling and analyzing hard data. His analysis, probably not surprisingly, shows that polling in aggregate over-estimated Democrats in each cycle. Needless to say, there’s a lot of “cope” in this analysis, reassuring fretful Republicans that it’s not as bad as it looks. But I was curious because this at least broadly matches with my observations. Not always but more often than not.
You can read the analysis here. You have to subscribe to his Substack to read the whole thing. But you can get the gist before you get to the paywall. I’ve reached out to several polling analysts whose judgment I trust to get their sense of the analysis. So perhaps I’ll have more to say based on any responses.
In any case, here are the main takeaways.
First, as I said, Senate polls have consistently overstated Democratic strength in the last four cycles. In a moment, we’ll get to why I think that’s the case.
Second: This applies in both presidential and midterm years but the details are different in the two kinds of national elections.
Three: In presidential years, the overstatement is biggest in the out-months and becomes more accurate by election day. But it still overstates Democratic strength by election day by 2.6 percentage points.
Four: In midterm elections, the overstatement is smaller, and by election day they only favor Democrats by one-third of a percentage point. So less overstatement and by election they’re pretty much on the nail.
(Edit Note: I was a little unclear in Ruffini’s account which numbers were all four cycles combined and which were the were the two presidentials and the two midterms. But the pattern here accurately reflects what he’s found if not the precise number.)
There are various other details in the analysis. But I think those are the key points.
In any case, as I said, the gist of the message is: “Don’t worry Republicans. It looks super bad right now. But this is the worst of it. It will start turning our way by the end of August.” But that doesn’t mean that the pattern isn’t accurate.
Ruffini doesn’t get into why this is the case much. My best guess is differential response: more social trust-oriented and engaged voters respond to polls more. And demographically those tend to be Democrats. It’s not an overwhelming difference but enough to show up as a small but at least broadly predictable bias. There’s also another more mundane explanation of the growing accuracy in the final months of the campaign. In the early months of a campaign, pollsters tend to poll “adults” or “registered voters”. They apply “likely voter” screens more aggressively in the final months and weeks as the contours of the electorate (who is going to vote) becomes more clear. You’d expect them to get more accurate, if we judge accuracy as the result on election day. Historically, “likely voter” filters favor Republicans, though that’s become less the case over the last decade as coalitions have shifted. This year, it seems to be the opposite.
In any case, as one of the experts I reached out to noted, in 2018 or 2022 if you’d gone into election day assuming a three- or four-point premium for the GOP you’d have totally missed the result since, in aggregate, the polls were very accurate. If there’s a big takeaway here it’s that recent cycles suggests the polls will tighten as we move toward election day — unless of course they don’t.
If you’re a Democrat, and if this analysis is accurate, it’s easy to feel a bit downcast about this: Oh, we can’t even trust the polls when they say we’re winning! Sometimes I see people reacting in a slightly different way: We can’t even deliver on our expected results!
This is definitely a wrongheaded way to look at things. Polling is just an artifact of the polling industry and mostly good-faith statistical weighting. It’s not like you should have done X but somehow choked in the final lap. The better approach is to factor in a small differential when thinking about races, especially in the out-months. It’s also important to remember that individual races will differ. Ruffini found that 70% of the races showed D-friendly polling errors, whereas you’d expect it to be 50% of the time if there was no systemic bias. That’s a big deal if it’s accurate. But it’s not all the time. In 30% of the cases the polls are overstating Republican odds. I was also curious to see the breakdowns in each cycle individually. But unless I missed it, he doesn’t provide those.
My own sense is that the national political conversation likely continues to underestimate the depth of Trump’s unpopularity and anti-incumbent mood generally. And that will see this on election day. But these patterns — assuming they’re broadly true — are important to bear in mind and broadly match with most of our lived experiences over the last decade.
