Member Newsletter
I wanted to share with you a few ideas, possible insights and caveats about campaign polls. These aren’t original to me in most cases, just some general points, observations, etc.
First, herding: Herding is the phenomenon in which even professional, good-faith polling operations start grouping together in the latter stages of a campaign because you don’t want to go too far out from the consensus numbers. Right now the national top lines have been between 2 and 4 points in Harris’ favor for a couple months. If you do a poll that gets you plus 10 in either direction, you’re going to think or are liable to think something’s wrong with your numbers. Somehow you’ve just got a spoiled set of data. Maybe you don’t release that poll or maybe you look again at the numbers and decide there are too few of some demographic subset and you re-weight that and it brings the topline back close to that 2-4 range.
It’s also the case that voter choice gets more settled in the final weeks of a campaign. So maybe the voters are actually herding themselves. There are lots of possibilities. But the general point here is that there are factors which can drive even ethical and professional pollsters in this herding direction.
It goes without saying that the more fly-by-nightish operations probably do this a lot. It’s sort of assumed that a lot of the junk pollsters let their freak flags fly for most of election season but then try to bring them in line with the herd in the final weeks to protect their reputations.
Implicit in the herding discussion above is what I’ve noted several times recently: the stability of this race is one of its most defining features. That could just be how settled the public mind is on this choice. It must be some of that. But I’ve heard another possibility proposed recently: Maybe pollsters are now modeling so aggressively that they’ve wrung all the movement out of the numbers?
Remember in earlier discussions I’ve noted that a poll is based not just on the 500 to 1,000 people polled but also on the pollster’s theory of the electorate. This is the art part of the art and science of polling. Modeling has become more aggressive as non-response has become a bigger and bigger problem. So you’re not just weighting men and women and Black people and white people but affluence, education levels, city vs. rural, how people say they voted last time, perhaps party identification. At a certain point you’re basically just saying you need 49% Trump supporters and 49% Harris supporters. I jest of course. But there could be something to this: maybe the race is so stable because the variability has been squeezed out through aggressive modeling. No one is proposing this as more than an idea, a “what if.” But it’s worth having in the back of our minds.
Next there’s an article you should read by Nate Cohn of the Times. It gets at a key methodological question about this year’s polling. Historically, it was always considered bad practice to weight polls based on how voters say they voted in the last election. Basically, people don’t always remember or tell the truth about how they voted. And it tends to pump up the advantage of the party that lost the last election. In other words, by traditional standards and accepted methodological considerations, by adding this voter recall weighting the pollsters have built a pro-Trump advantage into every result. But in an effort to more effectively avoid missing hidden Trump voters, most but not all pollsters decided to weight for voter recall for 2024. This isn’t obviously a bad idea, as Cohn explains. Polls did underestimate Trump in 2016 and 2020. And there are various ways pollsters have tried to account for the reasons why traditionally this isn’t a good idea to do. Cohn’s point and one I think he’s right to make is that that decision leads to two very different stories of the election. I’ve argued here and on the podcast that there’s as good a reason to think pollsters over-adjusted after 2024 as there is to expect Trump to again outdo the polls. Depending on how the race plays out it might be this specific decision that explains why one group or another of pollsters missed the mark.
Let me conclude by following up on that issue of whether pollsters may have over-adjusted after 2020. I’m an outsider to poll methodology. I understand the broad strokes. But in terms of evaluating what does or doesn’t make sense in methodological terms, I’m lost. But I do have one lingering question about 2020 and the fairly substantial poll miss that year. People forget that as shocking as the 2016 result was, the polling miss was quite small. I think it was about 1.5 points in the national numbers. That’s barely even a polling miss at all, really. The key was where it showed up in critical states. The 2020 miss was substantial, about 4 percentage points.
What people don’t focus on a lot is that 2020 was really sui generis for an election, both for polling and for voting. A substantial slice of the country was working from home right through 2020 and mailed in their votes so as not to risk their health going to the polls. Another slice tended to do the opposite as a matter of principle. Needless to say, that variance correlated highly with political identification. That is such a massive behavioral and sociological fact that it seems questionable to draw anything from it about how things work during a non-pandemic election. One theory I’ve heard is that possibly Democrats were just home a lot more than Republicans and more likely to take polls that year. Whether that was a partial driver of the polling error I don’t know. And one need be very wary of wishful thinking in a case like this. But like some of these other points, it’s worth keeping in mind. It seems to me to be a generally unassailable point that so many basic behaviors were different in 2020 and that they in many cases differed by party identification that it would be hard to confidently draw any lessons from it. In defense of the pollsters, they’re maybe not in a position to say, “2020 was weird. We want to see another election until we’re willing to say there’s a problem.” As I said, it’s something to have in the back of your head about polls this year, things that could suddenly seem like obvious explanations depending on whether polls miss or don’t miss the final result and whose favor they miss in if they do.