After the UK voted to leave the European Union a week ago, there was a lot of talk about how the polls blew the result, predicting a “Remain” win when in fact “Leave” managed a close but clear victory. The rejoinder to this criticism – I think a valid one – was that the balance of polls showed “Remain” in the lead but only barely. A better interpretation of the data was that the race was too close to call.
Beyond the statistical margin of error, there were aspects of the contest that made it difficult for pollsters to accurately model. If we combined statistical with what we might call methodological margin of error, the margin of the “Remain” lead was well within this combined margin of error.
But now TPM Reader DF sends me an email with a link to a study which puts the whole picture in a very different light – and even suggests, by implication, that Donald Trump might be doing better than we think.
Three public opinion experts, Harold D. Clarke, Matthew Goodwin, and Paul Whiteley, ran 121 Brexit polls through a statistical model intended to correct for methodological ‘artefacts’ and the ‘house effects’ of each pollster polling the race. After they ran the numbers through their model they came up with adjusted data that showed “Leave” was actually always ahead – sometimes by a bit and sometimes by a lot, but always ahead.
Here are their adjusted numbers.
Now, the model they used to analyze the polls is one I’m not fully able to explain or understand. But the notable point to me was what they came up with about online versus live phone polls (emphasis added).
Analysing the results (percentages of Remain and Leave supporters in various surveys) using our model reveals that both types of polls significantly overestimated the Remain vote share and underestimated the Leave share (see Figure 1). However, there were sizable differences between the two types of polls, with the mode effect for equaling 5.1 per cent for internet polls as compared to fully 9.2 per cent for telephone polls. Taken as a group, internet polls tended to perform considerably better than their telephone-based rivals.
There’s a bit of obscure technical jargon here. But the gist is both online and phone polls missed the mark. But phone polls were about twice as far off as Internet polls.
And here’s where we get to Trump.
There’s a clear pattern in the current national horse race numbers. Trump does better in the online polls than in what are generally considered the more prestigious national phone based polls conducted by major national publications like ABC/WAPO, NBC/WSJ, etc. Moreover, the Brexit “Leave” coalition had at least some similarities with Trump support in the US – generally older, more conservative, stronger in outlying or rural areas than in the major cities etc.
The theory here might be that online polls allow people to more openly voice what they believe to be socially unacceptable or less acceptable beliefs – pro-Brexit, pro-Trump etc. – something called “social desirability bias.”
But here’s the problem. Online polls consistently showed Trump stronger during the GOP primaries as well – indeed online polls showed an even greater Trump “bias” than they do now in the general. But when we could look back and see what actually happened in the voting it turned out that the online polls had considerably overstated Trump’s support. In the US, at this in this cycle and at least to date, online polls seem to be overstating Trump support, not compared to traditional phone polls but compared to actual vote results on election days.
In other words, in the US we see the same online poll advantage for the right-leaning, nationalist position as existed in the UK. Btu when people have actually voted, that advantage turned out to be illusory.
So what does it all add up to? I’m generally pro-online polls. Despite the reservations of more methodologically conservative statisticians, online polls have generally done well – and increasingly well – at the only real test of a campaign poll: the ability to accurately predict the outcome on election day. But there are many different approaches and many different standards of methodological consistency in online polls today. One cycle they’re better than live phone polls, another they’re worse. For now chalk this up as another data point in the on-going debate.
