Posts filed under Surveys (176)

January 15, 2016

Who got the numbers, how, and why?

The Dominion Post has what I’m told is a front page story about school costs, with some numbers:

For children starting state school this year, the total cost, including fees, extracurricular activities, other necessities, transport and computers, by the time they finish year 13 in 2028 is estimated at $35,064 by education-focused savings trust Australian Scholarship Group.

That increases to $95,918 for a child at a state-integrated school, and $279,807 for private school.

Given that the figures involve extrapolation of both real cost increases and inflation thirteen years into the future, I’m not convinced that a whole-education total is all that useful. I would have thought estimates for a single year would be more easily interpreted.  However, that’s not the main issue.

ASG do this routinely. They don’t have the 2016 numbers on their website yet, but they do have last year’s version. Important things to note about the numbers, from that link:

ASG conducted an online education costs survey among its members during October 2013. The surveys covered primary and secondary school. In all, ASG received more than 1000 survey responses.

So, it’s a non-random, unweighted survey, probably with a low response rate, among people signed up for an education-savings programme. You’d expect it to overestimate, but it’s not clear how much. Also

Figures have been rounded and represent the upper ranges that parents can reasonably expect to pay

‘Rounded’ is good, even though they don’t actually show much sign of having been rounded. ‘Represent the upper ranges’ is a bit more worrying when there’s no indication of how this was done — and when the Dom Post didn’t include this caveat in their story.

 

November 22, 2015

Helpful context

From the Herald

A study by sleep experts at Sealy UK found that those who kip on the right-hand side of the mattress are far more pessimistic than those who doze on the left.

 

Neil Robinson, Sealy’s top snooze analyst, said: “The research certainly highlights an interesting trend ” could it be possible that the left side of bed is the ‘right’ side?

 

November 16, 2015

Measuring gender

So, since we’re having a Transgender Week of Awareness at the moment, it seems like a good time to look at how statisticians ask people about gender, and why it’s harder than it looks.

By ‘harder than it looks’ I don’t just mean that it isn’t a binary question; we’re past that stage, I hope.  Also, this isn’t about biological sex — in genetics I do sometimes care how many X chromosomes someone has, but most questionnaires don’t need to know. It’s harder than it looks because there isn’t just one question.

The basic Male/Female binary question can be extended in (at least) two directions.  The first is to add categories to represent other ways people identify their gender beyond just male/female, which can be fluid over time, or can have more than two categories. Here a write-in option is useful since you almost certainly don’t know all the distinctions people care about across different cultures. In a specialised questionnaire you might even want to separate out questions about fluid/constant identity from non-binary/diversity, but for routine use that might be more than you need.

A second direction is to ask about transgender status, which is relevant for discrimination and (or thus) for some physical and mental health risks.  (Here you might want also want to find out about people who, say, identify as female but present as male.) We have very little idea how many people are transgender — it makes data on sexual orientation look really precise — and that’s a problem for service provision and in many other areas.

Life would get simpler for survey collectors if you combined these into a single question, or if you had a Male/Female/It’s Complicated question with follow-up questions for the third group. On the other hand, it’s pretty clear why trans people don’t like that approach. These really are different questions. For people whose answer to the first question is something like “it depends” or a culturally specific third option, the combination may not be too bad. The problem comes when answer to the second type of question might be “Trans (and yes I sometimes get comments behind my back at work but most people are fine)”, but the answer to the first “Female (and just as female as people with ovaries and a birth certificate, ok)”.

Earlier this year Stats New Zealand ran a discussion and  had a go at a better gender question, and it is definitely better than the old one, especially when it allows for multiple answers and for a write-in answer. They also have a ‘synonym list’ to help people work with free-text answers, although that’s going to be limited if all it does is map back to binary or three-way groups. What they didn’t do was to ask for different types of information separately. [edit: ie, they won’t let you unambiguously say ‘female’ in an identity question then ‘trans’ in a different question]

It’s true that for a lot of purposes you don’t need all this information. But then, for a lot of purposes you don’t actually need to know anything about gender.

(via Writehanded and Jennifer Katherine Shields)

November 9, 2015

Inelegant variation

These graphs are from the (US) National Cable & Telecommunications Association (the cable guys)

cableguy

Apart from the first graph, they are based on five-point agree-disagree scales, and show the many ways you can make pie and bar charts more interesting, especially if you don’t care much about the data. I think my favourites are the bendy green barchart-orbiting-a-black-hole and the green rectangles, where the bars disagree with the printed numbers.

Since it’s a bogus poll, using the results basically to generate artwork is probably the right approach.

November 6, 2015

Failure to read small print

smallprint

This story/ad/column hybrid thing on the Herald site is making a good point, that people don’t read the detailed terms and conditions of things. Of course, reading the terms and conditions of things before you agree is often infeasible — I have read the Auckland Transport HOP card T&Cs, but I don’t reread them to make sure they haven’t changed every time I agree to them by getting on a bus, and it’s not as if I have much choice, anyway.  When the small print is about large sums of money, reading it is probably more important.

The StatsChat-relevant aspect, though is the figure of $1000 per year for failing to read financial small print, which seemed strange. The quote:

Money Advice Service, a government-backed financial help centre in the UK, claims failure to read the small print is costing consumers an average of £428 (NZ$978) a year. It surveyed 2,000 consumers and found that only 84 per cent bothered to read the terms and conditions and, of those that did, only 17 per cent understood what they had read.

Here’s the press release (PDF) from Money Advice Service.  It surveyed 3000 people, and found that 84 per cent claimed they didn’t read the terms and conditions.

The survey asked people how much they believed misunderstanding financial terms in the last year had cost them. The average cost was £427.90.

So the figure is a bit fuzzier: it’s the average of what people reported believing they lost, which actually makes it more surprising. If you actually believed you, personally, were losing nearly a thousand dollars a year from not reading terms and conditions, wouldn’t you do something about it?

More importantly, it’s not failure to read the small print, it’s failure to understand it. The story claims only 17% of those who claimed to read the T&Cs thought they understood them — though I couldn’t find this number in the press release or on the Money Advice site, it is in the Mirror and, unsourced, in the Guardian.  The survey claims about a third misunderstood what ‘interest’  meant and of the 15% who had taken out a payday loan, more than half couldn’t explain what a ‘loan’ was, and one in five didn’t realise loans needed to be paid back.

As further evidence that either the survey is unreliable or that it isn’t a simple failure to read that’s the problem, there was very little variation between regions of the UK in how many people said they read the small print, but huge variation (£128-£1014in how much they said it cost them.

I’m not convinced we can trust this survey, but it’s not news that some people make unfortunate financial choices.  What would be useful is some idea of how often it’s really careless failure to read, how often it’s lack of basic education, how often it’s gotchas in the small print, and how often it’s taking out a loan you know is bad because the alternatives are worse.

November 1, 2015

Twitter polls and news feeds

aje

I don’t know why this feels worse that the bogus clicky polls on newspaper websites. Maybe it’s the thought of someone actually believing the sampling scheme says something useful. Maybe it’s being in Twitter, where following a news headline feed usually gets you news headlines. Maybe it’s that the polls are so bad: restricting a discussion of Middle East politics to two options with really short labels makes even the usual slogan-based dialogue look good in comparison.

In any case, I really hope this turns out to be a failed experiment, and that we can keep Twitter polls basically as jokes.

 

October 22, 2015

Second-hand bogus poll

Headline1 in 3 women watch porn – survey

Opening sentence: One in three young women regularly view porn, with many watching it on their smartphone, it has emerged.

It turns out this is “Some 31 per cent of participants in the survey by magazine Marie Claire.” If you Google, you can find the Marie Claire invitation to take the survey, with a link. There are also Facebook and YouTube versions of the invitation. It’s a self-selected internet survey; a bogus poll.

Considered in the context of its original purpose, this survey isn’t so bad. It’s part of a major project(possibly NSFW) by the magazine, and its contributing editor Amanda de Cadenet, to discuss women’s use of pornography.  The survey provided a way for them to involve readers, and a context for telling readers, however they responded, “there are lots of other women like you”.  From that point of view the quantitative unreliability and poorly-defined target population isn’t such a problem, though it would presumably be better to have the right numbers.

Disconnected from the magazine and presented as data-based news, the survey results have very little going for them.

October 19, 2015

Flag referendum stats

UMR have done a survey of preferences on the new flag candidates that can be used to predict the preferential-voting result.  According to their data, while Red Peak has improved a long way from basically no support in August, it has only improved enough to be a clear third to the two Lockwood ferns, which are basically tied for the lead both on first preferences and on full STV count.  On the other hand, none of the new candidates is currently anywhere near beating the current version.

The error in a poll like this is probably larger than in an election poll, because there’s no relevant past data to work with. Also, for the second round of the referendum, it’s possible that cutting the proposals down to a single alternative will affect opinion. And, who knows, maybe Red Peak will keep gaining popularity.

September 21, 2015

It’s bad enough without exaggerating

This UK survey report is being a bit loose with the details, in a situation where that’s not even needed

stem for boys

The survey of more than 4,000 girls, young women, parents and teachers, demonstrates clearly that there is a perception that STEM subjects and careers are better suited to male personalities, hobbies and brains. Half (51 percent) of the teachers and 43 percent of the parents surveyed believe this perception helps explain the low uptake of STEM subjects by girls. [emphasis added]

Those aren’t the same thing at all.  I believe this perception helps explain the low uptake of STEM subjects by girls. Michelle ‘Nanogirl’ Dickinson believes this perception helps explain the low uptake of STEM subjects by girls. It’s worrying that nearly more than half of UK teachers don’t believe this perception helps explain the low uptake of STEM subjects by girls.

On the other hand, this is depressing and actually does seem to be what the survey said:

Nearly half (47 percent) of the young girls surveyed said they believe such subjects are a better match for boys.

as does this

difficult subjects It would fit with NZ experience if a lot of boys felt the same about the difficulty of science and maths, but that wouldn’t actually make it any better.

 

September 8, 2015

Petitions and other non-representative data

Stuff has a story about the #redpeak  flag campaign, including a clicky bogus poll that currently shows nearly 11000 votes in support of the flag candidate. While Red Peak isn’t my favourite (I prefer Sven Baker’s Huihui),  I like it better than the four official candidates. That doesn’t mean I like the bogus poll.

As I’ve written before, a self-selected poll is like a petition; it shows that at least the people who took part had the views they had. The web polls don’t really even show that — it’s pretty easy to vote two or three times. There’s also no check that the votes are from New Zealand — mine wasn’t, though most of them probably are.  The Stuff clicky poll doesn’t even show that 11,000 people voted for the Red Peak flag.

So far, this Stuff poll at least hasn’t been treated as news. However, the previous one has.  At the bottom of one of the #redpeak stories you can read

In a Stuff.co.nz poll of 16,890 readers, 39 per cent of readers voted to keep the current flag rather than change it. 

Kyle Lockwood’s Silver Fern (black, white and blue) was the most popular alternate flag design, with 27 per cent of the vote, while his other design, Silver Fern (red, white and blue), got 23 per cent. This meant, if Lockwood fans rallied around one of his flags, they could vote one in.

Flags designed by Alofi Kanter – the black and white fern – and Andrew Fyfe each got 6 per cent or less of the vote

They don’t say, but that looks very much like this clicky poll from an earlier Stuff flag story, though it’s now up to about 17500 votes

flagpoll

You can’t use results from clicky polls as population estimates, whether for readers or the electorate as a whole. It doesn’t work.

Over approximately the same time period there was a real survey by UMR (PDF), which found only 52% of people preferred their favourite among the four flags to the current flag.  The referendum looks a lot closer than the clicky poll suggests.

The two Lockwood ferns were robustly the most popular flags in the survey, coming  in as the top two for all age groups; men and women; Māori; and Labour, National and Green voters. Red Peak was one of the four least preferred in every one of these groups.

Only 1.5% of respondents listed Red Peak among their top four.  Over the whole electorate that’s still about 45000, which is why an online petition with 31000 electronic signatures should have about the impact it’s going to have on the government.

Depending on turnout, it’s going to take in the neighbourhood of a million supporting votes for a new flag to overturn the current flag. It’s going to take about the same number of votes ranking Red Peak higher than the Lockwood ferns for it to get on to the final ballot.

In the Stuff story, Graeme Edgeler suggests “Perhaps if there were a million people in a march” would be enough to change the government’s mind. He’s probably right, though I’d say a million estimated from a proper survey, or maybe fifty thousand in a march should be enough. For an internet petition, perhaps two hundred thousand might be a persuasive number, if there was some care taken that they were distinct people and eligible voters.

For those of us in a minority on flag matters, Andrew Geddis has a useful take

In fact, I’m pretty take-it-or-leave-it on the whole point of having a “national” flag. Sure, we need something to put up on public buildings and hoist a few times at sporting events. But I quite like the fact that we’ve got a bunch of other generally used national symbols that can be appropriated for different purposes. The silver fern for putting onto backpacks in Europe. The Kiwi for our armed forces and “Buy NZ Made” logos. The Koru for when we’re feeling the need to be all bi-cultural.

If you like Red Peak, fly it. At the moment, the available data suggest you’re in as much of minority as me.