Think of a number and multiply it by 260,000,000
We haven’t had one of these for a while, but there’s some dodgy-looking extrapolation going on in the Keep New Zealand Beautiful litter audit. The audit itself is a good idea: measure litter in a detailed and reproducible way, so you can compare amounts now to amounts in the future and see whether things are getting greener and cleaner. And I don’t have any problems with how they conducted the survey.
But. The report (PDF) says (p18)
10,269,090,000 LITTERED CIGARETTE BUTTS polluting our ecosystem
and the Herald story says
Despite drops in smoking rates, discarded cigarette butts remained a big headache: some 10,269,090,000 were picked up, or 2,142 for every person in the country.
There weren’t 10 billion cigarette butts picked up. That would take a while. There were 39 cigarette butts picked up per 1000 square metres of land surveyed. With 10,000 477,000 square metres surveyed [update: I was confused by Table 2 in the report, which says 10,000 but is just illustrating the calculation (that’s the Table 2 on p25, not one of the others)], that comes to 390 18600. The detailed breakdowns in the report are fine, but there are also these extrapolations, where the amount of litter per 1000 square metres is scaled up by the number of 1000 square metre patches it would take to cover the whole country — about 260,000,000.
And, similarly, from the Herald quoting KNZB chief executive Heather Sanderson
“Extrapolated, that means 265,324,848 litres of illegal dumping – enough to fill 2,123 rail carriages, which if you stack them on top of each other, would be as high as 151 Sky Towers.”
is obtained by finding just under 1 litre (or 0.001 cubic metres) per 1000 square metres in the survey, and scaling up to the whole country. (Also, those imaginary rail carriages are being stacked end on end, which is probably not good for them)
Scaling up like this is how survey statistics works, but only if the sites you survey are an equal-probability random sample of the area The report doesn’t say they were, and it seems pretty unlikely, because it’s quite hard to get to a lot of randomly chosen bits of New Zealand, and these places — whether they’re up inaccessible mountains or in the middle of a big dairy farm — will tend to have less litter.
[Update: it’s obvious not an equal-probability sample of NZ; it could be some sort of stratified sample of the types of areas they were focusing on]
[Update, 13 September: Keep NZ Beautiful has modified the report to take out the dodgy extrapolations. Congratulations. The Herald hasn’t modified their story, though.]
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
OMG! There is only one person in my 10m2 office right now and that means that New Zealand’s population is… 26 billion people! No wonder I can’t get a park at Westfield Newmarket or get Spark streaming to work.
5 years ago
Chewing gum decades ago used to be the plague of public footpaths.
It hasn’t disappeared but now is uncommon. Did those little breath freshener caps replace them ?
As for the cigarette butts , Stats NZ says the main urban areas are 1.9% of landmass . It isnt all the smaller towns but lets try 2.5% of landmass, which would make 6.5 mill as the scaling factor so we are looking at 250 mill butts
5 years ago
That helps, but even within urban areas they oversampled places that were likely to have more litter. You’d really need land use information for the sampled areas and for NZ as a whole.
More importantly, though, estimating total litter load *wasn’t the point*. The litter audit wasn’t designed to allow that sort of estimation, and it shouldn’t be used that way. If they’d wanted national total litter load they would have used a different design.
5 years ago
Given that the point seems to be to facilitate across-time comparisons, even measuring the same “sample” plots each time would have limited value if the survey didn’t correct for changes to type and intensity of land use, type of litter, and mode of littering.
A vacant lot vs a public park on the same site might have very different patterns for static litter but relatively little change for wind-blown litter, whereas a shopping mall would all but eliminate wind-blown litter.
The litter loading on a bush trail would vary fairly directly with the number of people using the trail. Without changing the land-use designation, opening or closing its access points could make a vast difference to its litter.
Within a plot of 1000m² there might be a few small corners that are “secluded” and perhaps sheltered from the wind, that are likely places for smokers to “light up” — and consequently, more likely as places for butts to be discarded.
I wonder if dividing the sample area into only 10 plots makes it more difficult to make really meaningful comparisons in the future?
5 years ago
As someone who works for an organisation that produces a lot of statistics we have a process called the “sniff test’. That is, does the number smell right? If we get an “interesting” number often we look at other numbers that should look similar in pattern, magnitude or scale to see if we have a consistent statistical story.
For example we know that alcohol consumption is highly likely to be underestimated by respondents simply as – last time I looked – twice as much alcohol was available in NZ than our respondents claimed to consume.
So how about getting the figure for the supply end? My rough calculation is the figure corresponds to 500 million packets of cigarettes.
5 years ago
I thought about that, but I decided I didn’t really know how long cigarette butts might remain in the environment, or what proportion of them ended up as litter. It’s easy to see the number is wrong, but not so easy to see what might be a plausible number.
5 years ago
The downloadable report seems to have moved, to https://www.knzb.org.nz/wp-content/uploads/2019/09/KNZB-National-Litter-Audit-report-Online.pdf
(The link given above just says “page not found.)
The headline article is at https://www.knzb.org.nz/resources/research/nla/.
5 years ago
More importantly, they’ve taken out the dodgy extrapolations
5 years ago
It’s good that they’ve cleared the dodgy stats from the site, but in the absence of any more official retraction notice, pretty much no chance that the news stories that highlighted the stats will update or fix – or that the outlets will provide updated stories on it.
5 years ago
That reminds me, I should write about the latest disappearing Alzheimer’s cure, which failed in clinical trials back in March, and which has three
positive stories at the Herald and two at Stuff, and one each at Newshub and TVNZ, with no followup
5 years ago