Posts filed under Surveys (152)

December 29, 2014

What’s not in a name

I passed up this reprinted advertising-oriented survey story  about “The naughtiest names” the first time it came around. It’s back.

The findings come from a survey that looked at the names of more than 63,000 school children who logged good behaviour or achievement awards in online sticker books.

Those with the most good behaviour awards were named Jacob and Amy, closely followed by Georgia and Daniel.

Coincidentally, I’ve been listening to the BBC production of Good Omens, by Terry Pratchett and Neil Gaiman. It’s available online for the next three weeks. People who like that sort of  thing will find it’s the sort of thing they like. Early on, names are being suggested for a baby who turns out to be the Antichrist:

“Wormwood’s a nice name..Or Damien. Damien’s very popular….Or Cain. Very modern sound, Cain, really.”

This attempt to suggest ‘the naughtiest name’ failed dismally, and that’s probably true of the British survey as well.  The survey is probably a bit more representative of the population, but Good Omens is probably more realistic about the impact of names on the behaviour of children.

If you go to the original source, you see the originators of the survey didn’t really believe it either:

Neil Hodges, School Stickers Managing Director says, “The annual ‘Santa’s Naughty and Nice list’ is just a bit of fun, and obviously there are many Ella’s and Joseph’s that are perfect little angels, just as I’m sure there are many Amy’s and Jacobs that can be a bit of a handful.

though most of the mainstream media stories lost the disclaimer. This time it wasn’t the press release that was to blame.

It’s not that names have no effect. There’s a lot of research showing that identical job applications, for example, may be handled differently if different names are attached. There’s also a lot of social information in names — the story mentions research showing that you’re much more likely to get into Oxford or Cambridge if you’re called Eleanor than if you’re called Jade.

It’s possible there is some effect beyond social stratification and teacher prejudices, but this sort of survey is hopelessly unfit to reveal it.  That’s not the worst aspect, though. Even if the patterns of behaviour and name were real, they are soon going to be out of date. Patterns of first names change quite quickly, and this data presumably refers to kids who were named 5-10 years ago.  ‘Eleanor’ is now one of the names on the Naughty list.



December 22, 2014

How unrepresentative bogus polls can be

From @davejac on Twitter, clipped from Stuff



Since this is on the Census, we have good population data. If you include family-trust properties as ‘own’, which seems to be the intent, just over a quarter own mortgage-free, just under a third own but are paying a mortgage, and about a third are paying rent. The rest are more complicated.

The poll under-represents renters and over-represents owners, and it quite dramatically under-represents “Other”.

[update: those figures are for households, but the broad pattern of differences would be similar for people — there are more single-person households renting, but also large ones]

November 26, 2014

What doesn’t get into the papers

I complain a lot about the publicity-based surveys of varying quality that make it into the NZ media, but there’s a lot more that gets filtered out.

A journalist (who I’m not sure if I should name) sent me an example from Mitre 10

The research surveyed more than 1,500 New Zealanders on their connection to the quarter-acre dream and asked their opinions on the size of back yards and what they were doing to make the most of them.

An overwhelming 84 per cent of respondents agreed that they liked the idea of the traditional Kiwi quarter-acre paradise – a large plot of land with a standalone house on it, with plenty of room outdoors, and almost all said they would rather live on the traditional quarter-acre section than in high-density housing with reduced outdoor living spaces.

Over half of respondents felt that their outdoor living space is smaller now than what they had growing up (53%). Fifty percent of respondents attributed this to sections of land getting smaller, while 35 per cent believe houses are getting bigger, so there’s less room on a section for an outdoor living space.

The press release is a well-crafted example, with supporting evidence from QV that house sizes are increasing and quotes from a Massey University researcher — not about the survey, but about the general topic.

The survey, on the other hand, was fairly bogus. It was online, and most of the respondents got there through the Mitre 10 Facebook page.  You’d expect (and the Mitre 10 CEO has said) that the Facebook page attracts Mitre 10 customers, not necessarily a representative sample.  The report confirms this, with 88% of respondents being born in NZ, compared to about 75% of the population as a whole.

To make matters worse, here’s the reported data for the paragraphs quoted above. “Houses are bigger” and “sections are smaller” were alternative responses to the same question. You couldn’t answer that both were true — the correct answer, and the position that the report itself is pushing.



One more finding I can’t resist quoting: “The majority of Kiwis (24%) have spent between $1,000 and $5,000 on their outdoor living spaces over the past year. “

Untitled 2

October 20, 2014

Advertising about your weekend

Today’s Daily Mail story in the Herald is unusual, not because it’s a survey done to advertise a company, but because the company of that name in New Zealand is getting a freebie. The story is describes people lying about their boring weekends, and it’s a survey commissioned by Travelodge, the UK budget hotel chain. The hotel company with with the Travelodge brand in this part of the world is, as far as I can tell, not related.

What is notable about the story, which confused me at first when looking across multiple versions in the British media, is that it’s a re-run. Travelodge did the same survey in 2011, on a larger sample. Here’s the Mail story from last time; the Herald escaped it then.

The press release for this year’s survey isn’t up, but if it’s like the 2011 one it won’t give any information about how the survey was conducted, and only reports a few highlights of the results, so if it were about anything important you wouldn’t want to pay attention.

October 12, 2014

Unofficially over arithmetic

From the Herald (from the Washington Post), under the headline “Teens are officially over Facebook” (yes, officially)

Now, a pretty dramatic new report out from Piper Jaffray – an investment bank with a sizable research arm – rules that the kids are over Facebook once and for all, having fled Mark Zuckerberg’s parent-flooded shores for the more forgiving embraces of Twitter and Instagram.

This is based on a survey by Piper Jaffray, of 7200 people aged 13-19, (in the US, though the Herald doesn’t say that).

It looks as though US teens are leaving Facebook, but they sure aren’t flocking to Twitter, or, really, to Instagram. If you go to a story that gives the numbers, you see that reported Facebook use has fallen 27 percentage points. Instagram has risen only 7 percentage points, and Twitter has fallen by 4.


So, where are they going? They aren’t giving up on social media entirely — although “None” category wasn’t asked the first time around, it’s only 8 percent in the second survey.  It’s possible that teens are cutting down on the number of social media networks they use, but it seems more likely that the question was badly designed. Even I can think of at least one major site that isn’t on the list, Snapchat, which globalwebindex thinks is used by 42% of US internet-connected 16-19 year olds.

Incidentally: those little blue letters that look like they should be a link? They aren’t on the Herald site either, and on the Washington Post site they link to a message that basically says “no, not for you.”

October 8, 2014

What are CEOs paid; what should they be paid?

From Harvard Business Review, reporting on recent research

Using data from the International Social Survey Programme (ISSP) from December 2012, in which respondents were asked to both “estimate how much a chairman of a national company (CEO), a cabinet minister in a national government, and an unskilled factory worker actually earn” and how much each person should earn, the researchers calculated the median ratios for the full sample and for 40 countries separately.

The graph:



The radial graph exaggerates the differences, but they are already huge. Respondents dramatically underestimated what CEOs are actually paid, and still thought it was too much.  Here’s a barchart of the blue and grey data (the red data seems to only be available in the graph). Ordering by ideal pay ratio (rather than alphabetically) helps with the nearly-invisible blue bars: it’s interesting that Australia has the highest ideal ratio.


The findings are a contrast to foreign aid budgets, where the desired level of expenditure is less than the estimated level, but more than the actual level.  On the other hand, it’s less clear exactly what the implications are in the CEO case.


September 29, 2014

Stealth advertising survey

Stuff has a story that at first glance seems to be about baldness:

There are more men suffering hair loss in Auckland than anywhere in the country, with 42 per cent of those who live in the Super City thinning or completely bald.

In contrast, the hairiest region is Canterbury, where men are more rugged and sport glowing locks. Just 27 per cent of Canterbury men admit to suffering any hair loss.

We aren’t told the sample size or margin of error — if the survey was of 1000 people, you’d expect to get that sort of variation between the highest and lowest regions by chance.

I haven’t been able to find any more detailed results anywhere, but the important part of the story is actually in the next sentence

The headlining Colmar Brunton poll, commissioned by SRS Hair Clinic and released this week, surveyed men aged between 25 and 50.

That is, the point of this survey is to advertise a hair clinic (which sells a hair tonic that claims 100% Natural Ingredients and Zero Side Effects)


August 13, 2014

When are self-selected samples worth discussing?

From recent weeks, three examples of claims from self-selected samples:

In all three cases, you’d expect the pattern to generalise to some extent, but not quantitatively. The dating site in question specifically boasts about the non-representativeness of its members; the NZAS survey was sent to people who’d be likely to care, and there wasn’t much time to respond; scientists who had experienced or witnessed harassment would be more likely to respond and to pass the survey along to others.

I think two of these are worth presenting and discussing, and the other one isn’t, and that’s not just because two of them agree with my political prejudices.

The key question to ask when looking at this sort of probably non-representative sample, is whether the response you see would still be interesting if no-one outside the sample shared it. That is, the surveys tell us at a minimum

  • there exist 350 women in New Zealand who wouldn’t marry a man earning less than them, and are prepared to say so
  • there exist 200-odd scientists in NZ who think the National Science Challenges were badly chosen or conducted, and are prepared to say so
  • there exist 417 scientists who have experienced verbal sexual harassment, and 139 who have experienced unwanted physical contact from other research staff during fieldwork, and are prepared to say so.

I would argue that the first of these is completely uninteresting, but the second is contrary to the impressions being given by the government, and the third should worry scientists who participate in or organise fieldwork.


August 6, 2014

Income statistics

The Herald has a story headlined “Where to work if it’s money you’re after,” giving estimated median incomes across a range of job areas.  Sadly, if you read to the end, two of the sources are summaries of advertised salaries for advertised jobs on Seek and TradeMe.  That is, they are neither actual incomes, nor for the country as a whole.

Rather than just whinge about unrepresentative data, I looked at StatsNZ. They divide things up differently, so there was only one job group in the story that exactly matched one on NZ.Stat. People working in construction have a median weekly income of $840 and mean weekly income of $956 according to the NZ Income Survey. If most people in construction worked all year, without periods of unemployment, this would come to a median annual income of  $43,680 or a mean of $49,712.

The Herald thinks the median annual income in construction is $60,000-$78,000.



July 24, 2014

Infographic of the month

Alberto Cairo and pointed me to the infographic on the left, a summary of a residents’ survey from the town of Flower Mound, Texas (near Dallas/Fort Worth airport). The highlight of the infographic is the 3-D piecharts nesting in the tree, ready to hatch out into full-fledged misinformation.

At least, they look like 3-D pie charts at first glance.  When you look more closely, the data are three-year trends in approval ratings for a variety of topics, so pie charts would be even more inappropriate than usual as a display method.  When you look even more closely, you see that that’s ok, because the 3-D ellipses are all just divided into three equal wedges — the data aren’t involved at all.

flower_mound 2014 Citizen Survey Infographic_201407151504422733

The infographic on the right comes from the town government.  It’s much better, especially by the standards of infographics.

If you follow the link, you can read the full survey results, and see that the web page giving survey highlights actually describes how the survey was done — and it was done well.  They sent questionnaires to a random sample of households, got a 35% response rate (not bad, for this sort of thing) and reweighted it based on age, gender, and housing tenure (ie rent, own, etc) to make it more representative.  That’s a better description (and a better survey) than a lot of the ones reported in the NZ media.


[update: probably original, higher resolution version, via Dave Bremer.]