Posts filed under Surveys (156)

January 21, 2015

How to feel good about New Zealand

StatsChat criticises the NZ media a lot, but if you really want a target-rich zone, the place is the UK. Today, the Daily Express had this front page:


The biggest vote on this country’s ties to ­Brussels for 40 years saw 80 per cent say they no longer want to be in Europe, the ­Daily Express can reveal.

It marks a huge leap forward in this news­paper’s crusade to get Britain out of the EU.


This comes from a survey in three Conservative electorates in the southern UK (out of 650 electorates), where 100,000 questionnaires were distributed. About 12% said Britain should leave the EUK, about 3% were opposed, and the other 85% didn’t respond.

Other, better-conducted polling doesn’t find such a dramatic lead. Even a late-December poll by “Get Britain Out” found only 51% support for leaving the EU and consoled themselves by describing this as showing their campaign was gaining momentum.

(via @federicacocco)

January 20, 2015

Ask a silly question, get a silly answer

The monthly US FoodDemand survey added some questions about government policies this time around. Mostly these were reasonable (eg, do you support a tax on sugared sodas, which got 39% ‘Yes”, the same as here; do you support a ban on sale of marijuana, 46% yes)

However, one question was

“Do you support mandatory labeling for foods containing DNA?”

There’s no way this is a sensible question about government policies: it isn’t a reasonable policy or one that has been under public debate.  Most foods will contain DNA, the exceptions being distilled spirits, some candy, and (if you don’t measure too carefully) white rice and white flour. Nevertheless, 80% of people were in favour.

There was also a question “Do you support mandatory labeling for foods produced with genetic engineering”. This got 82% support.

It seems most likely that many respondents interpreted these questions as basically the same: they wanted labelling for food containing DNA that was added or modified by genetic engineering.  This isn’t what the researchers meant, since they write

A large majority (82%) support mandatory labels on GMOs, but curiously about the same amount (80%) also support mandatory labels on foods containing DNA.

If you ask a question that is nuts when interpreted precisely, but is basically similar to a sensible question, people are going to answer the question they think you meant to ask. People are helpful that way, even when it isn’t helpful.

January 6, 2015

Foreign drivers, again

The Herald has a poll saying 61% of New Zealanders want to make large subsets of foreign drivers sit written and practical tests before they can drive here (33.9%: people from right-hand drive countries; 27.4% everyone but Australians). It’s hard to tell how much of this is just the push effect of being asked the questions and how much is real opinion.

The rationale is that foreign drivers are dangerous:

Overseas drivers were found at fault in 75 per cent of 538 injury crashes in which they were involved. But although failure to adjust to local conditions was blamed for seven fatal crashes, that was the suspected cause of just 26 per cent of the injury crashes.

This could do with some comparisons.  75% of 538 is 403, which is about 4.5% of all injury crashes that year.  We get about 2.7 million visitors per year, with a mean stay of 20 days (PDF), so on average the population is about 3.3% short-term visitors.

Or, we can look at the ‘factors involved’ for all the injury crashes. I get 15367  drivers of motorised vehicles involved in injury crashes, and 9192 of them have a contributing factor that is driver fault (causes 1xx to 4xx in the Crash Analysis System). This doesn’t include things like brake failures.  So, drivers on average are at fault in about 60% of the injury crashes they are involved in.

Based on this, it looks as though foreign drivers are somewhat more dangerous, but that restricting them is very unlikely to prevent more than, say, 1-2% of crashes. If you consider all the ways we might reduce injury crashes by 1-2%, and think about the side-effects of each one, I don’t think this is going to be near the top of the list.

January 2, 2015

Maybe not a representative sample

The Dominion Post asked motorists why they thought the road toll had climbed, and what should be done about it.


Interestingly, three of the five(middle-aged, white, male ,Wellington area) motorists attributed it to random variation. That’s actually possible: the evidence for a real change in risk nationally is pretty modest (and the Wellington region toll is down on last year).

(via @anderschri5 on Twitter)

December 29, 2014

What’s not in a name

I passed up this reprinted advertising-oriented survey story  about “The naughtiest names” the first time it came around. It’s back.

The findings come from a survey that looked at the names of more than 63,000 school children who logged good behaviour or achievement awards in online sticker books.

Those with the most good behaviour awards were named Jacob and Amy, closely followed by Georgia and Daniel.

Coincidentally, I’ve been listening to the BBC production of Good Omens, by Terry Pratchett and Neil Gaiman. It’s available online for the next three weeks. People who like that sort of  thing will find it’s the sort of thing they like. Early on, names are being suggested for a baby who turns out to be the Antichrist:

“Wormwood’s a nice name..Or Damien. Damien’s very popular….Or Cain. Very modern sound, Cain, really.”

This attempt to suggest ‘the naughtiest name’ failed dismally, and that’s probably true of the British survey as well.  The survey is probably a bit more representative of the population, but Good Omens is probably more realistic about the impact of names on the behaviour of children.

If you go to the original source, you see the originators of the survey didn’t really believe it either:

Neil Hodges, School Stickers Managing Director says, “The annual ‘Santa’s Naughty and Nice list’ is just a bit of fun, and obviously there are many Ella’s and Joseph’s that are perfect little angels, just as I’m sure there are many Amy’s and Jacobs that can be a bit of a handful.

though most of the mainstream media stories lost the disclaimer. This time it wasn’t the press release that was to blame.

It’s not that names have no effect. There’s a lot of research showing that identical job applications, for example, may be handled differently if different names are attached. There’s also a lot of social information in names — the story mentions research showing that you’re much more likely to get into Oxford or Cambridge if you’re called Eleanor than if you’re called Jade.

It’s possible there is some effect beyond social stratification and teacher prejudices, but this sort of survey is hopelessly unfit to reveal it.  That’s not the worst aspect, though. Even if the patterns of behaviour and name were real, they are soon going to be out of date. Patterns of first names change quite quickly, and this data presumably refers to kids who were named 5-10 years ago.  ‘Eleanor’ is now one of the names on the Naughty list.



December 22, 2014

How unrepresentative bogus polls can be

From @davejac on Twitter, clipped from Stuff



Since this is on the Census, we have good population data. If you include family-trust properties as ‘own’, which seems to be the intent, just over a quarter own mortgage-free, just under a third own but are paying a mortgage, and about a third are paying rent. The rest are more complicated.

The poll under-represents renters and over-represents owners, and it quite dramatically under-represents “Other”.

[update: those figures are for households, but the broad pattern of differences would be similar for people — there are more single-person households renting, but also large ones]

November 26, 2014

What doesn’t get into the papers

I complain a lot about the publicity-based surveys of varying quality that make it into the NZ media, but there’s a lot more that gets filtered out.

A journalist (who I’m not sure if I should name) sent me an example from Mitre 10

The research surveyed more than 1,500 New Zealanders on their connection to the quarter-acre dream and asked their opinions on the size of back yards and what they were doing to make the most of them.

An overwhelming 84 per cent of respondents agreed that they liked the idea of the traditional Kiwi quarter-acre paradise – a large plot of land with a standalone house on it, with plenty of room outdoors, and almost all said they would rather live on the traditional quarter-acre section than in high-density housing with reduced outdoor living spaces.

Over half of respondents felt that their outdoor living space is smaller now than what they had growing up (53%). Fifty percent of respondents attributed this to sections of land getting smaller, while 35 per cent believe houses are getting bigger, so there’s less room on a section for an outdoor living space.

The press release is a well-crafted example, with supporting evidence from QV that house sizes are increasing and quotes from a Massey University researcher — not about the survey, but about the general topic.

The survey, on the other hand, was fairly bogus. It was online, and most of the respondents got there through the Mitre 10 Facebook page.  You’d expect (and the Mitre 10 CEO has said) that the Facebook page attracts Mitre 10 customers, not necessarily a representative sample.  The report confirms this, with 88% of respondents being born in NZ, compared to about 75% of the population as a whole.

To make matters worse, here’s the reported data for the paragraphs quoted above. “Houses are bigger” and “sections are smaller” were alternative responses to the same question. You couldn’t answer that both were true — the correct answer, and the position that the report itself is pushing.



One more finding I can’t resist quoting: “The majority of Kiwis (24%) have spent between $1,000 and $5,000 on their outdoor living spaces over the past year. “

Untitled 2

October 20, 2014

Advertising about your weekend

Today’s Daily Mail story in the Herald is unusual, not because it’s a survey done to advertise a company, but because the company of that name in New Zealand is getting a freebie. The story is describes people lying about their boring weekends, and it’s a survey commissioned by Travelodge, the UK budget hotel chain. The hotel company with with the Travelodge brand in this part of the world is, as far as I can tell, not related.

What is notable about the story, which confused me at first when looking across multiple versions in the British media, is that it’s a re-run. Travelodge did the same survey in 2011, on a larger sample. Here’s the Mail story from last time; the Herald escaped it then.

The press release for this year’s survey isn’t up, but if it’s like the 2011 one it won’t give any information about how the survey was conducted, and only reports a few highlights of the results, so if it were about anything important you wouldn’t want to pay attention.

October 12, 2014

Unofficially over arithmetic

From the Herald (from the Washington Post), under the headline “Teens are officially over Facebook” (yes, officially)

Now, a pretty dramatic new report out from Piper Jaffray – an investment bank with a sizable research arm – rules that the kids are over Facebook once and for all, having fled Mark Zuckerberg’s parent-flooded shores for the more forgiving embraces of Twitter and Instagram.

This is based on a survey by Piper Jaffray, of 7200 people aged 13-19, (in the US, though the Herald doesn’t say that).

It looks as though US teens are leaving Facebook, but they sure aren’t flocking to Twitter, or, really, to Instagram. If you go to a story that gives the numbers, you see that reported Facebook use has fallen 27 percentage points. Instagram has risen only 7 percentage points, and Twitter has fallen by 4.


So, where are they going? They aren’t giving up on social media entirely — although “None” category wasn’t asked the first time around, it’s only 8 percent in the second survey.  It’s possible that teens are cutting down on the number of social media networks they use, but it seems more likely that the question was badly designed. Even I can think of at least one major site that isn’t on the list, Snapchat, which globalwebindex thinks is used by 42% of US internet-connected 16-19 year olds.

Incidentally: those little blue letters that look like they should be a link? They aren’t on the Herald site either, and on the Washington Post site they link to a message that basically says “no, not for you.”

October 8, 2014

What are CEOs paid; what should they be paid?

From Harvard Business Review, reporting on recent research

Using data from the International Social Survey Programme (ISSP) from December 2012, in which respondents were asked to both “estimate how much a chairman of a national company (CEO), a cabinet minister in a national government, and an unskilled factory worker actually earn” and how much each person should earn, the researchers calculated the median ratios for the full sample and for 40 countries separately.

The graph:



The radial graph exaggerates the differences, but they are already huge. Respondents dramatically underestimated what CEOs are actually paid, and still thought it was too much.  Here’s a barchart of the blue and grey data (the red data seems to only be available in the graph). Ordering by ideal pay ratio (rather than alphabetically) helps with the nearly-invisible blue bars: it’s interesting that Australia has the highest ideal ratio.


The findings are a contrast to foreign aid budgets, where the desired level of expenditure is less than the estimated level, but more than the actual level.  On the other hand, it’s less clear exactly what the implications are in the CEO case.