Posts filed under Surveys (133)

March 18, 2014

Three fifths of five eighths of not very much at all

The latest BNZ-REINZ Residential Market Survey is out, and the Herald has even embedded the full document in their online story, which is a very promising change.

According to the report 6.4% of homes sales in March are  to off-shore buyers, 25% of whom were Chinese. 25% of 6.4% is 1.6%.

If you look at real estate statistics (eg, here) for last month you find 6125 residential sales through agents across NZ. 25% of 6.4% of 6125 is 98. That’s not a very big number.  For context, in the most recent month available, about 1500 new dwellings were consented.

You also find, looking at the real estate statistics, that last month was February, not March.  The  BNZ-REINZ Residential Market Survey is not an actual measurement, the estimates are averages of round numbers based on the opinion of real-estate agents across the country.  Even if we assume the agents know which buyers are offshore investors as opposed to recent or near-future immigrants (they estimate 41% of the foreign buyers will move here), it’s pretty rough data. To make it worse, the question on this topic just changed, so trends are even harder to establish.

That’s probably why the report said in the front-page summary “one would struggle, statistically-speaking, to conclude there is a lift or decline in foreign buying of NZ houses.”

The Herald  boldly took up that struggle.

March 4, 2014

What you don’t know

The previous post was about the failure of the ‘deficiency model’ of communication, which can be caricatured as the idea that people who believe incorrect things just need knowledge pills.

Sometimes, though, information does help. A popular example is that providing information to university students about the actual frequency of binge drinking, drug use, etc, can reduce their perception that ‘everyone is doing it’ and reduce actual risky  behaviour.

So, it’s interesting to see these results from a US survey about same-sex marriage

Regular churchgoers (those who attend at least once or twice a month), particularly those who belong to religious groups that are supportive of same-sex marriage, are likely to over- estimate opposition for same-sex marriage in their churches by 20 percentage points or more.

  • „„About 6-in-10 (59%) white mainline Protestants believe their fellow congregants are mostly opposed to same-sex marriage. However, among white mainline Protestants who attend church regularly, only 36% oppose allowing gay and lesbian people to legally marry while a majority (57%) actually favor this policy.
  • Roughly three-quarters (73%) of Catholics believe that most of their fellow congregants are opposed to same-sex marriage. However, Catholics who regularly attend church are in fact divided on the issue (50% favor, 45% oppose).

For survey nerds, the sampling methodology and complete questionnaire are also linked from that web page.

February 16, 2014

Most young Americans think astronomy is science

And they’re right.

The problem is they don’t know the difference between the words “astronomy” and “astrology”. So we get survey results like this,

A study released by the National Science Foundation finds nearly half of all Americans feel astrology—the belief that there is a tie between astrological events and human experiences—is “very” or “sort of” scientific. Young adults are even more prone to believe, with 58% of 18- to 24-year-olds saying it is a science.

Richard N. Landers, a psychologist in Virginia, thought the name confusion might be responsible, and ran a survey using Amazon’s Mechanical Turk, where you pay people to do simple tasks.  He asked people to define astrology and then to say whether they thought it was scientific.

What he found, is shown in this graph based on his data



People who think astrology is about horoscopes and predicting the future overwhelming don’t think it’s scientific — about 80% are in the ‘no’, and ‘hell, no’ categories. People who think astrology is about the solar system and stars  think it is pretty scientific or very scientific.

The data isn’t perfect — it’s from a much less representative sample than the NSF used — but there’s a very strong indication that confusion between “astronomy” and”astrology” could explain the otherwise depressing results of the NSF survey.


(via @juhasaarinen)

February 15, 2014

I only drink it for the pictures

Stuff has a story about differences in coffee preference between regions of NZ.

A customer survey published yesterday confirms the capital has the country’s biggest coffee snobs – almost three in four will choose an expensive brew over a less tasty one every time.

They don’t give enough information to work out how big the differences are, or how they compare to the uncertainty in the survey.

There’s slightly more information in the press release – still no uncertainty, but we are told that the figure is 72% for Wellington and 67% for the country as a whole. Not a terribly impressive difference, and almost certainly the survey isn’t large enough to be able to tell which region is really highest. Fortunately, it’s not a question where the true answer matters.

You have to go to the Canstar Blue site to find that the survey population isn’t Kiwis in general, or even coffee-drinkers, but people who have been to a chain coffee store at least once in the past six months.

Interestingly, although 17% to 33% of people (varying between regions) consider coffee alone to qualify as breakfast, only 10% to 14% say they drink coffee for the caffeine.

Yeah, right.

February 14, 2014

Interpreted with caution

There’s a new paper in The Lancet, summarising population-based surveys across the world that asked about non-partner sexual violence. The paper’s conclusion, from the abstract

Sexual violence against women is common worldwide, with endemic levels seen in some areas, although large variations between settings need to be interpreted with caution because of differences in data availability and levels of disclosure.

The story in Stuff has the headline Sexual assaults more than double world average, and starts

The rate of sexual assault in Australia and New Zealand is more than double the world average, according to a new report.

After several highly publicised rapes and murders of young women in India and South Africa, researchers from several countries  decided to review and estimate prevalence of sexual violence against women in 56 countries.

The results, published in the UK medical journal The Lancet, found that 7.2 per cent of women aged 15 years or older  reported being sexually assaulted by someone other than an intimate  partner at least once in their lives.

The study found that Australia and New Zealand has the third-highest rate, more than double the world average, with 16.4  per cent.

If you look at the raw numbers reported in the paper, they showed Australia/NZ at about ten times the rate of the Caribbean or southern Latin America or Eastern Europe, which is really not plausible. Statistical adjustment for differing types of survey reduced that margin, but as the researchers explicitly and carefully point out, a lot of the variation between regions could easily be due to variations in disclosure, and it suggests that rape is being underestimated in some areas.

As usual with extreme international comparisons, the headline is both probably wrong and missing the real point. The real point is that roughly one in six women in Australia & NZ report having experienced sexual violence.

January 17, 2014

Hard-to-survey populations

As we saw a few weeks ago with the unexpectedly high frequency of virgin births, it can be hard to measure rare things accurately in surveys, because rare errors will crowd out the true reports. It’s worse with teenagers, as a new paper from the Add Health study has reported. The paper is not open-access, but there’s a story in MedicalXpress.

So imagine the surprise and confusion when subsequent revisits to the same research subjects found more than 70 percent of the self-reported adolescent nonheterosexuals had somehow gone “straight” as older teens and young adults.

“We should have known something was amiss,” says Savin-Williams. “One clue was that most of the kids who first claimed to have artificial limbs (in the physical-health assessment) miraculously regrew arms and legs when researchers came back to interview them.”

This wasn’t just data-entry error, and it probably wasn’t a real change; after some careful analysis they conclude it was a mixture of genuine misunderstanding of the question (“romantic” vs “sexual” attraction), and, well, teenagers. Since not many teens are gay (a few percent), it doesn’t take many incorrect answers to swamp the real data.

It doesn’t matter so much that the number of gay and bisexual teenagers was overestimated. The real impact is on the analysis of health and social development in this population.  In Add Health and at least one other youth survey, according to the researchers, this sort of error has probably led to overestimating the mental and physical health problems of non-heterosexual teenagers.

January 14, 2014

Health food research marketing

The Herald has a story about better ways to present nutritional information on foods

“Our study found that those who were presented with the walking label were most likely to make healthier consumption choices, regardless of their level of preventive health behaviour,” Ms Bouton said.

“Therefore, consumers who reported to be unhealthier were likely to modify their current negative behaviour and exercise, select a healthier alternative or avoid the unhealthy product entirely when told they would need to briskly walk for one hour and 41 minutes to burn off the product.

“The traffic light system was found to be effective in deterring consumers from unhealthy foods, while also encouraging them to consume healthy products.”

This sounds good. And this is a randomised experiment, which is an excellent feature.

However, it’s just an online survey of 591 people, about a hypothetical product, so what it actually found was that the labelling system was effective in deterring people from saying they would buy unhealthy foods, encouraging them to say they would consume healthy products and made them more likely to say they would exercise. That’s not quite so good. It’s a lot easier to get people to say they are going to eat better, exercise more, and lose weight that to get them to actually do it.

Another interesting feature is that this new research has appeared on the Herald website before. In October 2012 there was a story based on the first 220 survey responses

Not only were people more likely to exercise when they saw such labels, they also felt more guilty, Ms Bouton said.

“My findings showed that the exercise labelling was significantly more effective in both chocolate and healthier muesli bars in encouraging consumers to exercise after consumption.

“It increased the likelihood of having higher feelings of guilt after consumption and was more likely to stop [the participant] consuming the chocolate bar with the exercise labelling.”

The 2012 story still didn’t raise the issue of what people said versus actual behaviour, but it did get an independent opinion, who pointed out that calories aren’t the only purpose of food labelling.

More importantly, the stories and the two press releases are all the information I could find online about the research. There don’t seem to be any more details either published or in an online report. It’s good to have stories about scientific research, and this sort of experiment is an important step in thinking about food labelling, but the stories are presenting stronger conclusions that can really be supported by a single unpublished online survey.

December 20, 2013

Best? Worst? It’s all just numbers

From the Herald, based on a survey by a human resource company that’s lost its shift key

“New Zealand lags behind other Asia-Pacific countries in wage equality”

From the Ministry of Women’s Affairs, based on the Quarterly Employment Survey

New Zealand’s gender pay gap is the equal lowest in the OECD (along with Ireland). The gender pay gap at 10.1 percent (2013) is the lowest in the Asia-Pacific region.

It’s not that Herald personnel don’t know about the Government figures: about a month ago they ran a story describing the small but worrying increase in the governments estimate of the pay gap.

Now, it could be that both these things are true — perhaps they define the pay gap in different ways, or maybe the gap is much larger in some industries (and, necessarily, smaller in others).  But if you’re going to run a dramatically different estimate of such an important national statistic, it would be helpful to explain why it is different and how it was estimated, and say something about the implications of the difference.

Especially as once you find the company on the web (quite hard, since their name is “font”), you will also find they run an online salary survey website that provides self-reported salaries for a self-selected sample.  I hope that isn’t where the gender pay gap information is coming from.


December 19, 2013

Difficulties in interpreting rare responses in surveys

If some event is rare, then your survey sample won’t have many people who truly experienced it, so even a small rate of error or false reporting will overwhelm the true events, and can lead to estimates that are off by a lot more than the theoretical margin of sampling error.

The Herald has picked up on one of the other papers (open access, not linked) in this year’s Christmas BMJ, which looks at data from the National Longitudinal Study of Youth, in the US. This is an important social and health survey, and the paper is written completely seriously. Except for the topic

Of 7870 eligible women, 5340 reported a pregnancy, of whom 45 (0.8% of pregnant women) reported a virgin pregnancy (table 1). Perceived importance of religion was associated with virginity but not with virgin pregnancy. The prevalence of abstinence pledges was 15.5%. The virgins who reported pregnancies were more likely to have pledged chastity (30.5%) than the non-virgins who reported pregnancies (15.0%, P=0.01) or the other virgins (21.2%, P=0.007).


A third group of women (n=244) not included in analysis, “born again virgins,” reported a history of sexual intercourse early in the study but later provided a conflicting report indicating virginity. Reports of pregnancy among born again virgins were associated with greater knowledge of contraception methods with higher failure rates (withdrawal and rhythm methods) and lower interview quality (data not shown), and reports from this group may be subject to greater misclassification error.

The survey had carefully-designed and tested questions, and used computer-assisted interviewing to make participants more willing to answer potentially embarrassing questions. It’s about as good as you can get. But it’s not perfect.

December 15, 2013

He knows if you’ve been bad or good

From today’s Herald (or the very similar story at 3News)

A Wellness in the Workplace survey show sickies taken by people who aren’t really ill are estimated to account for 303,000 lost days of work each year, at a cost of $283 million.

Skipping over the estimate of over $900/day for the average cost of a sickie, this is definitely an example where a link to the survey report and some description of methodology might be helpful. The report says

The survey was conducted during the month of
June 2013. In total, 12 associations took part,
sending it out to a proportion of their members.
In addition, BusinessNZ sent the questionnaire
to a number of its Major Companies Group
members. Respondents were asked to report their
absence data for the 12-month period 1 January to
31 December 2012 and provide details of their policies
and practices for managing employee attendance.

In total, 119 responses were received from entities
across the private and public sectors.

which gives more idea about potential (un)representativeness. But most importantly,  while the survey has real data on numbers of absences and on policies, the information on how likely employees were to take sick leave when not sick was just the opinion of their employers. Unless you work for Santa or the NSA, this is going to have a large component of guesswork.

If you’re an employer, and you want to know whether inappropriate use of sick leave is a problem for your organisation, do you want to rely on your own guesses, or on an average of guesses by an anonymous assortment of 119 other organisations around the country?