WaPo published a fascinating piece this morning tied to their latest poll of Kansas. The headline, figurative if not literal, is that they came up with a result that shows incumbent Republican Sen. Roger Marshall not behind Adam Hamilton, but way behind. But they don’t buy their own result. What’s fascinating and laudable is that polls like this, this late in the cycle, often never get released or they get adjusted to bring them more into line with pollsters’ expectations or the consensus of other polls. But here the Post made the decision to release the poll as is and pull back the veil on all the internal details and decisions they made to get the poll as close to what they think is accurate as they could.
Differential response has been the bane of pollsters for more than a decade. We know that no one answers the phone (or the text message) when pollsters get in touch anymore. The problem is that Republicans and Democrats “never” respond at slightly different rates. And even more specifically, the slice of usually disaffected voters that Donald Trump has repeatedly activated when he’s on the presidential ballot are the most poll-resistant of all. So a huge amount of what has happened in the polling industry over the last decade is pollsters trying to find ways to tweak numbers in some coherent way to adjust for getting too few of those poll-shy Trump voters. (And to be clear, this isn’t the old idea that they’re embarrassed to say they support Trump. They just don’t want to talk to pollsters.) Further complicating the issue is that that activation — and the polling misses tied to it — doesn’t seem to happen in midterm elections, based on 2018 and 2022.
As you can see in this chart, the problem the Post faced with differential response has been manageable in most races this year. Polls in Michigan and Maine differed by around 4% points. But in Kansas Democrats were wildly more likely to respond to the polls. So the Post did the standard weighting to bring the numbers into line with the state population — weighting for partisanship, geography, various demographic markers, etc. But after they did all that they still came up with a result they didn’t find credible, or at least that they had significant doubts about. Those results were Hamilton 52%, Marshall 43% and a third party candidate at 4%.
Adding to this was respondents’ recall of how they voted in 2024. This is actually another source of potential error but in the other direction. After the sample was weighted, respondents in aggregate remembered voting for Donald Trump by 7%. In fact he won the state by 16%. This part in the write-up was obscure to me because I couldn’t tell just which way and at which stage they were trying to adjust for voters’ selective memories. So for instance you have this: “Post-Schar School polls typically weight by 2024 vote choice. But even after doing so, registered independents in the poll reported voting for Harris over Trump by 25 percentage points, compared with a nearly even split in a 2024 AP VoteCast survey released after the election.”
The problem with vote recall is that you don’t know what it means. If your sample says they voted for Donald Trump by 10% and the actual vote was 20%, maybe you have a sample of the electorate that is less Trumpy than the electorate at large. But given how unpopular Trump has become, maybe they’re not telling the truth about who they voted for. (This doesn’t have to be willful lying; people have selective memories.) In practice it’s probably a combination of both. But which it is or the precise mix can tilt the poll into error in different directions.
We get some additional insight on this incredibly difficult question from our friend Lakshya Jain, now at The Argument. They did a livecast last night where they discussed their latest and final poll (11 point Democrat spread on the generic ballot). The key is that they have an ongoing panel. So they can talk to the same identified people — literally the same individuals — and compare who they said they voted for right after the 2024 election and who they say they voted for in that election now. And a substantial number of them have gone from saying they voted for Trump to now saying Harris. I can’t remember the exact percentage. But point being, this isn’t some hypothetical thing. Most pollsters are weighting to partisan recall. But not everyone is an accurate narrator.
As long as we’re deep in this forest of polling error hypotheticals, I wanted to mention one other. G. Elliott Morris is another big player in the polling and public opinion analysis field. He’s generally been among those in this cycle saying we don’t really have any reason to think polling error will go one way or another this cycle. And that we’re wrong to dismiss the possibility that polls could actually be underestimating Democratic strength. But he’s also been scanning the numbers asking the following question: if there is yet another systematic underestimation of Republicans, where is most likely to come from?
He came up with one possibility which he discussed in a post this week. In the weighting above, pollsters weight for all sorts of things: gender, education, vote recall, urban/rural — and a bunch of other things. But Morris notes that they don’t usually cross reference these things. And what he came up with is that his poll was oversampling Trump voters who live in blue urban areas as opposed to rural red areas. And it turns out that matters. Because urban Trumpers are defecting at higher rates than rural ones. So you can weight to make sure you have the right number of Trump voters and the right number of rural voters. But if you don’t combine the two things you’re missing some of the most loyal Trumpers. The impact wasn’t huge. Morris found after crunching a lot of numbers that it generated between .5 and 1 percentage point difference in the generic ballot.
That’s meaningful but not game changing. But there’s a further hitch. These are also the type of voters who’ve tended not to vote when Trump isn’t on the ballot. There’s also decent evidence that these voters are demotivated. So that’s another reason why they might not show up. Non-response can signal the same kind of low motivation or disenchantment that leads especially occasional voters not to vote. Or maybe this will happen and it will be offset by one of the several possible ways that polls could be underestimating Democratic strength. Morris points specifically to the dissembling voter recall issue I noted above as one of those possibilities.
In any case, none of this discussion is really an attempt to figure out just what to make of that single poll. It’s just a fascinating window into this moment in the world of polling in which a mix of technological and sociological transformations have made the whole enterprise totally different and quite difficult to do. In terms of the Kansas race, there are two other polls of the race released in the last 24: CNN (Hamilton +4) and Emerson (Hamilton +2). Those can give us perhaps a more realistic view of the race, though I can’t help but wonder whether similar discussions were going on with those pollsters and they resolved the questions differently than the Post.