Posts written by Thomas Lumley (2645)

The Herald has a story that, I think, satisfies all the best-practice guidelines for scientific news.

They are reporting on new Ministry of Health cancer mortality statistics for 2009, and trends up to that year. In particular

• The source of the information is described well
• They quote both total deaths and rates per 100,000 population, and explain why the trends are different.
• They ask experts for comment and context, and give reasonable explanations of what might be causing the trends

A couple of things I would add to the story:  first, the mortality rates per 100,000 population are also age-adjusted, so they take into account the aging population, and second, an important contributing factor to the reduction in the cancer mortality rate over time is the reduction in smoking.  Improvements in diagnosis and treatment also help, but they are not the whole story, especially for lung cancer.

If you want the actual numbers, they are here.

The Herald has a fairly good story about a poll on motorway tolls:  the target population (Auckland City), the sample size, the results, and the sampling method are all described.

The sample wasn’t a random sample, but according to the description, it was still a reasonable sample

Those surveyed were drawn from a research panel recruited by Horizon in accordance with the demographics of Auckland’s adult population at the last Census, weighted to match age, gender, personal income, employment and education levels

The respondents will be a biased sample, but the pollers have made efforts to correct this bias.  That’s what is done in good-quality national surveys even if a random sample is taken, because non-response inevitably makes the sample less representative than it should be.

There are two points that would have been welcome in the story which weren’t there.  Firstly, there is no margin of error.  Using a biased sample and reweighting will typically give a larger margin of error than using a random sample, though not so large as to make a big difference to the conclusions the Herald reports.  Secondly, the respondents were asked for their opinions about different ways of raising 10 billion dollars to pay for major city projects, but it’s not clear whether they were asked about just not doing the projects and saving the 10 billion dollars.

From the Herald this morning

More people have died in fewer car smashes since January 1 than at this time last year, prompting a Government reminder about the responsibility drivers hold over others’ lives.

“The message for drivers is clear,” Associate Transport Minister Simon Bridges said yesterday of a spate of multi-fatality crashes that have boosted the road toll to 161.

The number of fatal crashes is 133, compared to 144 last year at this time, and the number of deaths is 161, compared to 155 last year.

How do we calculate how much random variation would be expected in counts such as these?  It’s not sampling error in the sense of opinion polls, since these really are all the crashes in New Zealand.  We need a mathematical model for how much the numbers would vary if nothing much had changed.

The simplest mathematical model for counts is the Poisson process.  If dying in a car crash is independent for any two people in NZ, and the chance is small for any person (but not necessarily the same for different people) then number of deaths over any specified time period will follow a Poisson distribution.    The model cannot be exactly right — multiple fatalities would be much rarer if it were — but it is a good approximation, and any more detailed model would lead to more random variation in the road toll than the Poisson process does.

There’s a simple trick to calculate a 95% confidence interval for a Poisson distribution, analogous to the margin of error in opinion polls.  Take the square root of the count, add and subtract 1 to get upper and lower bounds, and square them: a count of 144  is consistent with underlying averages rates from 121 to 169.   And, as with opinion polls, when you look at differences between two years the range of random variation is about 1.4 times larger.

Last year we had an unusually low road toll, well below what could be attributed to random variation.  It still isn’t clear why, not that anyone’s complaining.  The numbers this year look about as different from last year’s as you would expect purely by chance.  If the message for drivers is clear, it’s only because the basic message is always the same:

Stuff has a headline “Living by the sea is good for your health” (with the page title “Life’s a beach”). The story starts

Sick people have been sent to the seaside for centuries and now a study has proven that living by the coast is good for your health.

Let’s see how it rates against the guidelines

(more…)

These are from the UK Science Media Centre, part of its submission to the Leveson Inquiry

• State the source of the story – e.g. interview, conference, journal article, a survey from a charity or trade body, etc. – ideally with enough information for readers to look it up or a web link.
•  Specify the size and nature of the study – e.g. who/what were the subjects, how long did it last, what was tested or was it an observation? If space, mention the major limitations.
• When reporting a link between two things, indicate whether or not there is evidence that one causes the other.
• Give a sense of the stage of the research – e.g. cells in a laboratory or trials in humans – and a realistic time-frame for any new treatment or technology.
•  On health risks, include the absolute risk whenever it is available in the press release or the research paper – i.e. if ’cupcakes double cancer risk’ state the outright risk of that cancer, with and without cupcakes.
•  Especially on a story with public health implications, try to frame a new finding in the context of other evidence – e.g. does it reinforce or conflict with previous studies? If it attracts serious scientific concerns, they should not be ignored.
•  If space, quote both the researchers themselves and external sources with appropriate expertise. Be wary of scientists and press releases over-claiming for studies.
• Distinguish between findings and interpretation or extrapolation; don’t suggest health advice if none has been offered.
•  Remember patients: don’t call something a ’cure’ that is not a cure.
• Headlines should not mislead the reader about a story’s contents and quotation marks should not be used to dress up overstatement

This blog would be a lot less interesting (except to rugby fans) if the media followed these guidelines.  I think the second-last item is the only one that hasn’t been the basis for one or more posts.     (via)

You will have heard about the terrible shootings in Colorado.

From a post by Kieran Healy, at Crooked Timber, responding to the tragedy: death rates from assault, per 100,000 population per year, for the US and 19 other OECD countries.  New Zealand is roughly in the middle (his post gives separate plots for each country).  Dots are the data for individual years, the curves are smoothed trends with margin of error.

The much higher rate in the US is obvious, but so is the decline.

Part of the decline is attributable to better medical treatment, so that assault victims are less likely to die, but far from all of it.  The rate of reports of aggravated assault is also down over the same time period.  Similarly, simple explanations like gun availability probably contribute but can’t explain the whole pattern.

The decline in violent deaths is so large that it shows up in life expectancy comparisons.  New York, and especially Manhattan, used to have noticeably worse life expectancy than Boston, but the falling rate of violent deaths and the improvements in HIV treatment now put Manhattan, and the rest of New York City, at the top of US life expectancy

Surveys are not perfect: some people misunderstand the question, some people recall incorrectly, some responses are written down incorrectly by the poller, and some people just lie.   These biases happen in both directions, but their impact is not symmetrical.

Suppose you had a survey that asked “Have you ever been abducted by aliens?”  We can be sure that false ‘Yes’ results will be more common than false ‘No’ results, so the survey will necessarily overestimate the true proportion. If you wrote down the wrong answer for 1% of people, you’d end up with an estimate that was 1% too high.

In principle, the same issue  could be a serious problem in estimating the support for minor parties: about 1% of people voted for ACT at the last election, and 99% didn’t.  Suppose you poll 10000 people and ask them if they voted for ACT, and suppose that 100 of them really were ACT voters. If your opinion poll gets the wrong answer, randomly, for 1% of people, you will get the wrong answer from 1 of the true ACT voters, and 99 of the true non-ACT voters, so you will report 100+99-1=198 ACT voters and 9900+1-99 = 9802 non-ACT voters.  You would overestimate the votes for ACT by a factor of two!  Keith Humphreys, who we have linked to before, postulates that this is why US polls indicating support for a third-party candidate tend to seriously overestimate their support.

I’m skeptical.  Here in NZ, where we really have minor parties, there is no systematic tendency to overestimate the support they receive.  ACT got 1% of the vote, and that was close to what the polls predicted. Similarly, the Maori Party, and the Greens received about the same number of votes in the last election as averages of the polls had predicted.  For NZ First, the election vote was actually higher than in the opinion polls.  Similarly, for the Dutch general election in 2010 there was pretty good agreement between the last polls and the election results.  Even in Australia, where there is effectively a two-party system in the lower house (but with preferential voting), the opinion poll figures for the Greens agreed pretty well with the actual vote

It’s true that measurement error tends to bias towards 50%, and this matters in some surveys, but I would have guessed the US phantom third party support is the result of bias, not error. That is, I suspect people tend to overstate their support for third-party candidates in advance of the election, and that in the actual election they vote strategically for whichever of the major parties they dislike least.   My hypothesis would imply not much bias in countries where minor-party votes matter, and more bias in countries with first-past-the-post voting.  Unfortunately there’s also a pure measurement error hypothesis that’s consistent with the data, which is that people are just more careful about measuring minor-party votes in countries where they matter.

So.  The 2012 Global Innovation Index is out and NZ looks quite good.  Our only Prime Minister has a graph on his Facebook page that looks basically like this.

The graph shows that NZ was at rank 28 in 2007 and is now at rank 13.

A bar chart for two data points is a bit weird, though not nearly as bad as the Romney campaign’s efforts at Venn diagrams in the US.

The scaling is also a bit strange.  The y-axis runs from 1 to 30, but there’s nothing special about rank 30 on this index. If we run the y-axis all the way down to 141 (Sudan), we get the second graph on the right, which shows that New Zealand, compared to countries across the world, has always been doing pretty well.

Now, there are some years missing on the plot, and the Global Innovation Index was reported for most of them.  Using the complete data, we get a graph like

So, in fact, NZ was doing even better on this index in 2010, and we get some idea of the year-to-year fluctuations.   Now, a barchart is an inefficient way to display data with just one short time series like this: a table would be better.

More important, though, what is this index measuring.  Mr Key’s Facebook page doesn’t say. Some of the commenters do say, but incorrectly (for example, one says that it’s based on current government policies).  In fact, the  exact things that go into the index change every year.  For example, the 2012 index includes Wikipedia edits and Youtube uploads,  in early years internet access and telephone access were included.  There are also changes in definitions: in early years, values were measured in US$, now they are in purchasing-power parity adjusted dollars.

Some of the items (such as internet and telephone access) are definitely good, others (such as number of researchers and research expenditure) are good all things being equal, and for others (eg, cost of redundancy dismissal in weeks of pay, liberalised foreign investment laws) it’s definitely a matter of opinion.Some of the items are under the immediate control of the government (eg public education expenditure per pupil, tariffs), some can be influenced directly by government (eg, gross R&D funding, quality of trade and transport infrastructure), and some are really hard for governments to improve  in the short term (rule of law, GMAT mean test score, high-tech exports, Gini index).

Since the content and weighting varies each year, it’s hard to make good comparisons. On the plus side, the weighting clearly isn’t rigged to make National look good — the people who come up with the index couldn’t care less about New Zealand — but the same irrelevance will also tend to make the results for New Zealand more variable.   Some of the items in the index will have been affected by the global financial crisis and the Eurozone problems. New Zealand will look relatively better on these items, for reasons that are not primarily the responsibility of the current governments even in those countries, let alone here.

I’d hoped to track down why New Zealand had moved up in the rankings, to see if it was on indicators that the current administration could reasonably take credit for, but the variability in definitions makes it very hard to compare.

Stuff has a story about twins in Canterbury, which is driven by two general human tendencies shared even by statisticians: thinking babies are cute, and overestimating the strangeness of coincidences.  We hear that

Canterbury mums have given birth to 21 sets of twins in the past six weeks.

and

 10 years ago the average would have been about six to eight sets a month.

Using the StatsNZ Infoshare tool (go to Population | Births -VSB | Single and multiple confinements by DHB) we find about 100 sets of multiple births per year in Canterbury DHB and a further dozen or so in South Canterbury DHB, without much change for the past ten years.  That means about nine or so multiple births per month on average.  If you use the average twin rate for all of NZ  (2.7%) and the number of births in the Canterbury region, you get a slightly lower 7.7 sets of twins per month on average.

If there are, on average, 9 multiple births per month, how long would you have to wait for a six-week period with 21?  Because the possible six-week periods overlap, it’s hard to do this calculation analytically, but we can simulate it: 9 per month is 108 per year, which is 108/52 per week.  We simulate a long string of one-week counts from a Poisson distribution with mean 108/52, and see how long we have to  wait between six-week totals of at least 21.  The average waiting time is about two years.  (you have to be a bit careful: the proportion of six-week intervals over 21 is a lot more than one in two years, because of the overlap between six-week intervals)

So, this is a once in two years coincidence if we just look at Canterbury.  It’s much more likely if twin stories from other regions might also end up as news — the probability is hard to estimate, because twins in Canterbury really are more newsworthy than in, say, Waikato.

The American Statistical Association gives an annual award for Excellence in Statistical Reporting.  This year it goes to Amanda Cox, a graphics editor at the New York Times, some of whose graphs we’ve highlighted on the blog.  Here are some more examples and talks.

The award was created to encourage and recognize members of the communications media who have best displayed an informed interest in the science of statistics and its role in public life. The award can be given for a single statistical article or for a body of work. In selecting the recipient, consideration is given to:

• Correctness, clarity, fairness, brevity, and professionalism of the communication
• Importance, relevance and overall effectiveness in impacting the intended audience
• Impact on the growth and national or regional exposure of statistics
• Appreciation and emphasis of the statistical aspects of a particular issue or event
• Excellent coverage of research on statistics or statistical issues