Polls aren’t dead yet
There’s a new paper in the journal Nature Human Behaviour analysing a huge collection of election poll data: over 30,000 polls. The researchers’ conclusion is straightforward: polls have not become less accurate. Unfortunately, all the nice graphs are behind a paywall. Fortunately, the data isn’t, and I can draw you a nice graph of my own
The graph shows all the poll results back to 1970, split up into panels by how many weeks before the election they were. I’m showing just one party per poll: the data have conveniently been coded so it’s a big party (eg Labour for NZ, Conservatives for UK). Each panel shows the error in the poll plotted against the year of the election; the red line is an average.
The red lines are basically flat. Despite cellphones, the internet, political polarisation, and millennials, average polling error hasn’t changed all that much over the past fifty years.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
What’s the absolute error in this case? The | proportion voting for each party at the election minus the proportion voting for that party in the poll | ?
6 years ago
Yes, (well, the absolute value of that).
6 years ago
In most cases a nationwide poll may be ‘accurate’ but isnt measuring what the
same thing as election is doing. Ask Clinton about that.
In US, electoral college decides the election and UK its numbers of MPs elected in mini races. Last UK result had conservatives losing seats despite increasing vote by 5.5%. NZ is lucky in that we have a single nationwide list and numbers of Mps proportional to that.
6 years ago
That is indeed true. It’s quite hard to predict a close UK election for that reason. Even in NZ, if a party is close to the 5% threshold it’s hard.
However, in addition to that, lots of people have been saying in recent years that polls are getting systematically less accurate *as predictors of the final vote*. That’s what this paper addresses: they haven’t.
6 years ago
IDK – the absolute error seems to be getting more variable over time – on average, they’re as accurate as always but if you have a wildly inaccurate survey then it’s more likely to be now-ish.
6 years ago
That’s mostly a visual artifact from having more polls nowadays
6 years ago
OK, so I downloaded the data and had a bit of hack. I restricted my data to within 50 days of the election and before 2014. The latter because the amount of data drops away from 2014 which may create biases.
I qqnormed the errors (not absolute errors) for each decade from 1940 to (partial) 2010s.
For all the decades up to and including the 90’s the data fits pretty much on a straight line with a few little misfits at the end. In the 00’s and 10’s the data has quite marked kinks in the tails.
And here are the means and variances for each decade:
40’s 0.3 15
50’s 0.1 15
60’s -0.3 20
70’s 0.3 16
80’s 0.0 12
90’s 0.0 12
00’s -0.3 13
10’s -0.8 29
So the 00’s similar in their variance even if the data isn’t normal but the restricted 10’s data has a much larger variance.
It’s not the last word because there are a whole range of explanatory factors that could be influencing things e.g. going to proportional representation in NZ. But at face value the graphs look like something is happening now-ish. And with an error of over 30 percentage points something is very wrong somewhere.
6 years ago