Posts filed under Polls (117)

May 24, 2016

Microplummeting

Headline: “Newshub poll: Key’s popularity plummets to lowest level”

Just 36.7 percent of those polled listed the current Prime Minister as their preferred option — down 1.6 percent — from a Newshub poll in November.

National though is steady on 47 percent on the poll — a drop of just 0.3 percent — and similar to the Election night result.

So, apparently, 0.3% is “steady” and 1.6% is a “plummet”.

The reason we quote ‘maximum margin of error’, even though it’s a crude summary, not a good way to describe evidence, underestimates variability, and is a terribly misleading phrase, is that it at least gives some indication of what is worth headlining.  The maximum margin of error for this poll is 3%, but the margin of error for a change is 1.4 times higher, about 4.3%.

That’s the maximum margin of error, for a 50% true value, but it doesn’t make that much difference– I did a quick simulation to check. If nothing happened, the Prime Minister’s measured popularity would plummet or soar by more than 1.6% between two polls about half the time purely from sampling variation.

 

April 28, 2016

Marking beliefs to market

Back in August, I wrote

Trump’s lead isn’t sampling error. He has an eleven percentage point lead in the poll averages, with sampling error well under one percentage point. That’s better than the National Party has ever managed. It’s better than the Higgs Boson has ever managed.

Even so, no serious commentator thinks Trump will be the Republican candidate. It’s not out of the question that he’d run as an independent — that’s a question of individual psychology, and much harder to answer — but he isn’t going to win the Republican primaries.

Arguably that was true: no serious commentator, as far as I know, did think Trump would be the Republican candidate.  But he is going to win the Republican primaries, and the opinion polls haven’t been all that badly wrong about him — better than the experts.

March 11, 2016

Getting to see opinion poll uncertainty

Rock’n Poll has a lovely guide to sampling uncertainty in election polls, guiding you step by step to see how approximate the results would be in the best of all possible worlds. Highly recommended.

Of course, we’re not in the best of all possible worlds, and in addition to pure sampling uncertainty we have ‘house effects’ due to different methodology between polling firms and ‘design effects’ due to the way the surveys compensate for non-response.  And on top of that there are problems with the hypothetical question ‘if an election were held tomorrow’, and probably issues with people not wanting to be honest.

Even so, the basic sampling uncertainty gives a good guide to the error in opinion polls, and anything that makes it easier to understand is worth having.

poll-land

(via Harkanwal Singh)

February 21, 2016

Evils of axis

From One News, tweeted by various people:

CbtrS2MUEAALlrk

The y-axis label is wrong: this has nothing to do with change, it’s percent support.

The x-axis label is maximally unhelpful: we can guess that the most recent poll is in February, but what are the earlier data? You might think the vertical divisions are months, but the story says the previous poll was in October.

Also, given that the measurement error is large compared to the expected changes, using a line graph without points indicating the observations is misleading.

Overall, the graph doesn’t add to the numbers on the right, which is a bit of a waste.

December 19, 2015

Punk’d

Earlier this year a current affairs program announced that they would have an interview with the man who didn’t get swallowed by a giant anaconda. Taken literally, this doesn’t restrict the options much.  There’s getting on for three billion men who haven’t been swallowed by giant anacondas; you probably know several yourself.  On the other hand, everyone knew which guy they meant.

There’s a branch of linguistics, called ‘pragmatics’, that studies how everyone knows what you mean in cases like this. The “Cooperative Principle” and Grice’s Maxims look at the assumption that everyone’s trying to move the conversation along and isn’t deliberately trolling.

One of the US opinion polling companies, Public Policy Polling, seems to make a habit of trolling its respondents.  This time, they asked whether people were in favour of bombing Agrabah.  30% of Republican supporters were. So were 19% of Democratic supporters, though for some reason this has been less widely reported. As you know, of course, since you are extremely well-read, Agrabah is not a town or region in Syria, nor is it held by Da’esh. It is, in fact, the fictional location of Disney’s Aladdin movie, starring among others the late, great Robin Williams.

I’m pretty sure that less than 30% even of Republican voters really support bombing a fictional country. In fact, I’d guess it’s probably less than 5%. But think about how the question was asked.  You’re a stereotypical Republican voter dragged away from quiet dinner with your stereotypical spouse and 2.3 stereotypical kids by this nice, earnest person on the phone who wants your opinion about important national issues.  You know there’s been argument about whether to bomb this place in the Middle East. You can’t remember if the name matches, but obviously if they’re asking a serious question that must be the place they mean. And it seemed like a good idea when it was explained on the news. Even the British are doing it. So you say “Support”.

The 30% (or 19%) doesn’t mean Republicans (or Democrats) want to bomb Aladdin. It doesn’t even mean they want to bomb arbitrary places they’ve never heard of. It means they were asked a question carefully phrased to sound as if it was about a genuine geopolitical controversy and they answered it that way.

When Ali G does this sort of thing to political figures, it’s comedy. When Borat does it to unsuspecting Americans it’s a bit dubious. When it’s mixed in with serious opinion polling, it risks further damaging what’s already a very limited channel for gauging popular opinion.

December 11, 2015

Against sampling?

Stuff has a story from the Sydney Morning Herald, on the claim that smartphones will be obsolete in five years. They don’t believe it. Neither do I, but that doesn’t mean we agree on the reasons.  The story thinks not enough people were surveyed:

The research lab surveyed 100,000 people across its native Sweden and 39 other countries.

With around 1.9 billion smartphone users globally, this means ConsumerLab covered just 0.0052 per cent of active users for its study.

This equates to about 2500 in each country; the population of Oberon

If you don’t recognise Oberon, it’s a New South Wales town slightly smaller than Raglan.

Usually, the Sydney Morning Herald doesn’t have such exacting standards for sample size. For example, their recent headline “GST rise backed by voters if other taxes cut: Fairfax-Ipsos poll” was based on 1402 people, about the population of Moerewa.

The survey size is plenty large enough if it was done right. You don’t, as the saying goes, have to eat the whole egg to know that it’s rotten. If you have a representative sample from a population, the size of the population is almost irrelevant to the accuracy of survey estimates from the sample. That’s why opinion polls around the world tend to sample 1000-2000 people, even though that’s 0.02-0.04% of the population of New Zealand, 0.004%-0.009% of the population of Australia, or 0.0003-0.0006% of the population of the USA.

What’s important is whether the survey is representative, which can be achieved either by selecting and weighting people to match the population, or by random sampling, or in practice by a mixture of the two.  Unfortunately, the story completely fails to tell us.

Looking at the Ericsson ConsumerLab website, it doesn’t seem that the survey is likely to be representative — or at least, there aren’t any details that would indicate it is.  This means it’s like, say, the Global Drug Survey,  which also has 100,000 participants, out of over 2 billion people worldwide who use alcohol, tobacco, and other drugs, and which Stuff  and the SMH have reported on at great length and without the same skepticism.

December 8, 2015

What you do know that isn’t so

The Herald (and others) are reporting an international Ipsos-Mori poll on misperceptions about various national statistics.  Two of the questions are things I’ve written about before: crude wealth inequality and proportion of immigrants.

New Zealanders on average estimated that 37% of our population are immigrants.  That’s a lot — it’s more than New York or London. The truth is 25%, which is still higher than most of the other countries. Interestingly, the proportion of immigrants in Auckland is quite close to 37%, and a lot of immigration-related news seems to focus on Auckland.   I think the scoring system based on absolute differences is unfair to NZ here: saying 37% when the truth is 25% doesn’t seem as bad as saying 10% when the truth is 2% (as in Japan).

We also estimated that 1% of the NZ population own 50% of the wealth. Very similar estimates came from a lot of countries, so I don’t think this is because of coverage of inequality in New Zealand.  My guess is that we’re seeing the impact of the Credit Suisse reports (eg, in Stuff), which say 50% of the world’s wealth is owned by the top 1%.  Combined with the fact that crude wealth inequality is a bogus statistic anyway, the Credit Suisse reports really seem to do more harm than good for public knowledge.

September 28, 2015

Seeing the margin of error

A detail from Andrew Chen’s visualisation of all the election polls in NZ:

polls

His full graph is somewhat interactive: you can zoom in on times, select parties, etc. What I like about this format is how clear it makes the poll-to-poll variability.  The poll result for, say, National isn’t a line, it’s a cloud of uncertainty.

The cloud of uncertainty gets narrower for minor parties (as detailed in my cheatsheet), but for the major parties you can see it span an entire 10-percentage-point grid cell or more.

September 10, 2015

Do preferential voting and similar flags interact badly?

(because I was asked about Keri Henare’s post)

Short answer: No.

As you know, we have four candidate flags. Two of them, the Lockwood ferns, have the same design with some colour differences. Is this a problem, and is it particularly a problem with Single Transferable Vote (STV) voting?

In the referendum, we will be asked to rank the four flags. The first preferences will be counted. If one flag has a majority, it will win. If not, the flag with fewest first preferences will be eliminated, and its votes allocated to their second-choice flags. And so on. Graeme Edgeler’s Q&A on the method covers the most common confusions. In particular, STV has the nice property that (unless you have really detailed information about everyone else’s voting plans) your best strategy is to rank the flags according to your true preferences.

That’s not today’s issue. Today’s issue is about the interaction between STV and having two very similar candidates.  For simplicity, let’s consider the extreme case where everyone ranks the two Lockwood ferns together (whether 1 and 2, 2 and 3, or 3 and 4). Also for simplicity, I’ll assume there is a clear preference ranking — that is, given any set of flags there is one that would be preferred over each of the others in two-way votes.  That’s to avoid various interesting pathologies of voting systems that aren’t relevant to the discussion. Finally, if we’re asking if the current situation is bad, we need to remember that the question is always “Compared to what?”

One comparison is to using just one of the Lockwood flags. If we assume either that there’s one of them that’s clearly more popular, or that no-one really cares about the difference, then this gives the same result as using both the Lockwood flags.

Given that the legislation calls for four flags this isn’t really a viable alternative. Instead, we could replace one of the Lockwood flags with, say, Red Peak.  Red Peak would then win if a majority preferred it over the remaining Lockwood flag and over each of the other two candidates.  That’s the same result that we’d get adding a fifth flag, except that adding a fifth flag takes a law change and so isn’t feasible.

Or, we could ask how the current situation compares to another voting system. With first-past-the-post, having two very similar candidates is a really terrible idea — they tend to split the vote. With approval voting (tick yes/no for each flag) it’s like STV; there isn’t much impact of adding or subtracting a very similar candidate.

If it were really  true that everyone was pretty much indifferent between the Lockwood flags or that one of them was obviously more popular, it would have been better to just take one of them and have a different fourth flag. That’s not an STV bug; that’s an STV feature; it’s relatively robust to vote-splitting.

It isn’t literally true that people don’t distinguish between the Lockwood flags. Some people definitely want to have black on the flag and others definitely don’t.  Whether it would be better to have one Lockwood flag and Red Peak depends on whether there are more Red Peak supporters than people who feel strongly about the difference between the two ferns.  We’d need data.

What this argument does suggest is that if one of the flags were to be replaced on the ballot, trying to guess which one was least popular need not be the right strategy.

September 9, 2015

Assessing popular opinion

One of the important roles played by good-quality opinion polls before an election is getting people’s expectations right.  It’s easy to believe that the opinions you hear everyday are representative, but for a lot of people they won’t be.  For example, here are the percentages for the National Party for each polling place in Auckland Central in the 2014 election. The curves show the margin of error around the overall vote for the electorate, which in this case wasn’t far from the overall for the whole country.

kael

For lots of people in Auckland Central, their neighbours vote differently than the electorate as a whole.  You could do this for the whole country, especially if the data were in a more convenient form, and it would be more dramatic.

Pauline Kael, the famous New York movie critic, mentioned this issue in a talk to the Modern Languages Association

“I live in a rather special world. I only know one person who voted for Nixon. Where they are I don’t know. They’re outside my ken. But sometimes when I’m in a theater I can feel them.”

She’s usually misquoted in a way that reverses her meaning, but still illustrates the point.

It’s hard to get hold of popular opinion just from what you happen to come across in ordinary life, but there are some useful strategies. For example, on the flag question

  • How many people do you personally know in real life who had expressed a  preference for one of the Lockwood fern flags and now prefer Red Peak?
  • How many people do you follow on Twitter (or friend on Facebook, or whatever on WhatsApp) who had expressed a  preference for one of the Lockwood fern flags and now prefer Red Peak?

For me, the answer to both of these is “No-one”: the Red Peak enthusiasts that I know aren’t Lockwood converts. I know of some people who have changed their preferences that way — I heard because of my last StatsChat post — but I have no idea what the relevant denominator is.

The petition is currently just under 34,000 votes, having slowed down in the past day or so. I don’t see how Red Peak could have close to a million supporters.  More importantly, anyone who knows that it does must have important evidence they aren’t sharing. If the groundswell is genuinely this strong, it should be possible to come up with a few thousand dollars to get at least a cheap panel survey and demonstrate the level of support.

I don’t want to go too far in being negative. Enthusiasm for this option definitely goes beyond disaffected left-wing twitterati — it’s not just Red pique — but changing the final four at this point really should require some reason to believe the new flag could win. I don’t see it.

Opinion is still evolving, and maybe this time we’ll keep the Australia-lite flag and the country will support something I like next time.