November 25, 2012

Clarity begins at home

Stuff’s story on the World Giving Index would have been better just using it as a hook for a discussion of charities in NZ, but they couldn’t stop themselves from referring to details. Which turn out to be wrong.

For example

The World Giving Index, which compares countries’ charitable behaviour in giving money, time and helping a stranger, found New Zealand was slightly less giving than in 2010 when New Zealand and Australia were found to be the most generous, with 57 per cent of people doing some charitable work.

It’s not 57% of people (unless you think those behaviours are independent).  The index value of 57 is the average of the proportions for the three things that the survey actually measured.  The value of 57 is exactly the same as in the previous survey.  NZ went down in the ranking because other countries increased their Giving Index value.

Fundraising Institute of New Zealand chief executive James Austin is quoted as saying

“When you start boiling it down, even though they are very sophisticated in the way they put their statistics together, everything from exchange rates has an impact on it,”

Either he wasn’t asked a question about the World Giving Index, or he doesn’t understand it either. This isn’t an inventory of actual monetary sums given. It’s just an average of three percentages: % who volunteered time, % who gave money, % who helped a stranger. Exchange rates really don’t come into it.

The story also doesn’t mention the cautionary note in the Australasia section of the World Giving Index report

 It is important to note that these surveys were conducted before devastating floods crippled Queensland, Australia in January 2011, and the tragic earthquake that struck New Zealand in February 2011, so any change in giving behaviour after these disasters is not captured in this year’s analysis.

That’s useful context for the poll-based claim in the story

A Sunday Star-Times poll of 763 readers reflected the World Giving Index, with 53 per cent of respondents having changed the way they donate in the past 12 months. More than half of those who had changed their habits admitted to donating less money.

So, this poll actually had very different findings from the World Giving Index survey, possibly because it was asking completely different questions, but possibly because it was about a non-overlapping period of time.

And we haven’t even got to margins of error or sampling bias.
avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »