Posts written by Thomas Lumley (1962)


Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient

March 23, 2017

This ain’t exactly real

From Matthew Beveridge on Twitter, who also has a post on reporting standards.


In a poll of 1000 people, if we make the (generous) assumption that it’s a uniform random sample, these minor-party results mean 8 people for The Opportunities Party, 7 for the Māori Party, 4 for ACT and 4 for UnitedFuture.  That’s not a very impressive lead.

A 95% confidence interval (the equivalent of the usual margin of error statement, using my cheatsheet)  for TOP would be 0.3% to 1.6%; for the Māori Party 0.3% to 1.4%, and for ACT and UnitedFuture 0.1 to 1.0%.  There’s a lot of overlap.   A 95% confidence interval for the ratio of TOP to ACT support goes from 0.63 to 7.5, showing that more-sophisticated analysis confirming the conclusion: ACT could be more popular than TOP, or much less popular, but we don’t have enough information to be sure. Obviously, the same thing will be true only more so for the TOP/Māori comparison.

But there’s a more important problem: for at least three of these parties, the party vote is relatively unimportant. The Māori Party [correction] has won one  list seat in its history, and neither ACT and UnitedFuture has for the past two elections.  What matters most for these parties is their support in their key electorates.  It’s hard to poll for electorate support, both because it’s hard to sample from electorates and because you have to distinguish party vote and electorate vote intentions reliably. Judging from past election campaigns, we’ll probably get some polling of the Māori electorates, but it’s unlikely we’ll get any useful polling of Ōhāriu and Epsom.  However, I would feel pretty safe in predicting that ACT will end up with a seat, and much less confident that TOP will.

Democracy is coming

We have an election this year, so we are starting to have polling.

To save time, here are some potentially useful StatsChat posts about election polls:

  • A simple cheatsheet for working out the margin of error for minor parties (also including a simple Excel macro)
March 21, 2017

Believing surveys

There was a story on Stuff yesterday claiming that 85% of Kiwis brush their teeth too hard, based on a survey by a company that sells soft toothbrushes. The survey involved over 1000 people, and that’s about all we know, except that the reported rate of twice-per-day brushing was about 15 percentage points higher than in the 2009 NZ Oral Health Survey.

Mark Hanna tweeted about the press release by survey issue


I want to expand on this.  Why is research, especially survey research, different?  When David Fisher interviews people involved with gambling addiction, we don’t need anything more than his story. When Fletcher Building says they are cutting $150m from their profit estimates, we don’t need anything more than their press release. So why isn’t enough that this toothbrush company says they have a survey?

The issue is responsibility.  If an investigative journalist reports statements from people, it’s the journalist’s reputation that makes those reports credible. If there are anonymous sources, again it’s the reputation of the journalist and the newspaper that makes us believe the sources really exist and are their claims are credible.  When a company says it’s introducing a new product or is revising its income estimates, the company is the only authoritative source of information, and the claims are treated as mere claims by the company, not as facts.

With a survey press release, the journalist typically isn’t vouching for the correctness of the interpretation or the validity of the methodology; that’s not their expertise.  And we can’t tell from the toothbrush story whether it was a real survey, or a well-calibrated online panel, or whether it was just a bogus clicky poll on a website somewhere. There’s no attribution, and there’s no responsibility. Even so, the claims don’t get treated as mere advertising; they get reported in basically the same way as all research findings.

If we’re going to treat a survey of this sort as showing anything,  and if the journalist isn’t vouching for the information, the minimum standard is that we can find out what was done.  The company doesn’t need to nerd up its press release with details, but they can put them on a website somewhere — how they found people, what the response rate was, something about who they sampled, what the actual questions were. Or, if their survey was done by a reputable market research firm, tell us that, and at least we know someone who understands the issues is standing behind the claims.

March 20, 2017

A blood test for autism?

There’s a story on Radio NZ Nine to Noon on a blood test for autism that is supposedly 96% accurate, and on what its implications might be for earlier specialised care of kids on the autism spectrum. As you’d expect from Radio NZ, the questions about the implications of a test were good.  The interview also raised the question of whether the test, developed on kids aged 3-10, would work equally well on younger children where the benefit is potentially greater.

What got missed out on a little was the meaning of “96% accurate”.  In the research study, 81 of 83 children on the autism spectrum and 73 of 76 neurotypical kids with no history of behavioral or neurologic abnormalities were correctly diagnosed, which is very impressive. But that’s not how the test would be used. In practice you might be screening the whole population, or screening kids with relatives on the autism spectrum, or diagnosing kids where ASD is suspected.

For whole-population screening, the 3.9% false-positive rate is more of a problem.  Based on current US statistics, roughly 1.5% of children are on the autism spectrum. So, of 1000 kids tested, about 15 would be correctly diagnosed and about 39 would be false positives. Now, it could be that the benefit to the 15 is much larger than the harm to the 39 and the test is worthwhile, but if we were going to have policy discussions about the test in that context, “96% accuracy” wouldn’t be a helpful way to describe it.

For children who have relatives with ASD you’d expect it to be harder to separate out ‘neurotypical’ from not. And that’s what the research shows:


The red and blue curves show the variation in blood test results for unrelated neurotypical and autism-spectrum children. The yellow shows the results for neurotypical siblings of children on the autism spectrum: there’s a lot less separation, and more tests will be wrong. Again, the test might be accurate enough to be useful, but “96% accuracy” is misleading.

Perhaps the way the test could be most helpful is as a step in diagnosis for children whose behaviour makes parents or doctors suspect ASD. There, the proportion on the autism spectrum might be more similar to the 50% in the study. However, we’ve currently got no idea how accurate the test is in such children; no-one has looked.

March 18, 2017


  • A guy tracked all the words his son learnt in the first 20 months, and made graphs.
  • Based on survey data, Peter Beinart argues in the Atlantic “those who don’t regularly attend church are more likely to suffer from divorce, addiction, and financial distress.” Fred Clark (who does regularly attend church) has a different suggestion “It’s also entirely possible — and quite likely once you allow yourself to think about it — that all of these things make church-going more difficult and less likely.” 
  • From the archive that now has online, Mark Broatch interviews Eula Bliss about her book on vaccination.
  • Apps that use your phone to monitor your health or nudge you towards better behaviour have obvious potential.  One of the first evaluations of an app baseed on Apple’s ResearchKit shows that the potential isn’t matched by actual. “only 131 participants took at least a week’s worth of surveys and a six-month milestone survey. That’s 1.7 percent”. Story at Ars Technica, research paper
  • The new has a good story on the history and uptake of a treatment for premature babies that came from New Zealand research on sheep.
  • Mark Hanna has an interactive graphic comparing how often NZ Police use different tactical options for stopping people, by ethnicity.
  • Does it matter how long you’ve already waited for a bus? In New York, not really.
  • Nat Dudley talks about colour-blindness and accessibility of graphics.

Two cheers for genomics

PCSK9 inhibitors are one of the high-profile stories of genomics in medical research.  The gene’s function was previously unknown: it was identified as important for cholesterol metabolism by genetic studies.  People with mutations that increase the activity of the protein have high LDL cholesterol; people with mutations that destroy the activity have low LDL cholesterol. And, importantly, there’s at least one person walking around and alive and healthy with mutations breaking both her copies of the gene, so inhibiting it looked relatively safe.  It was an obvious target for drug development and a showcase for the benefits of large-scale genetic research.

Three drug companies have made injectable antibodies that block the activity of PCSK9 and dramatically lower LDL cholesterol. One dropped out last year because their drug got attacked by the patients’ immune systems.  We’re now seeing the first results of clinical trials looking at whether the LDL cholesterol reduction leads to fewer heart attacks.

From a New Zealand point of view, the results are mostly of theoretical interest. Pharmac isn’t likely to subsidise these treatments for large groups of people any time soon.  However, we do still care what the trials show, because they help answer some questions about cholesterol. The research paper is here; two good commentaries on it are here and here

Amgen’s drug, evolocumab, reduced LDL cholesterol by about two-thirds (from an average of 2.3 mmol/l to 0.78 mmol/l). The combined rate of heart attack, stroke, and death from heart disease was 20% lower; there was only a 15% reduction in the longer shopping list of bad events that the study put its money on for the primary analysis.

So, first, lowering LDL cholesterol by a different mechanism from statins has also resulted in lower heart attack rates.  That reinforces the evidence that LDL cholesterol really matters; it’s not just a marker like smoke from a fire.  Given the largely failed efforts to improve health with drugs that raise HDL cholesterol, this is good to know. Second, lowering LDL cholesterol this way seems fairly safe. There wasn’t any detectable harm (apart from the localised symptoms of the injections themselves). There could be rarer or more subtle effects, of course.

And finally, while the results are qualitatively positive, the actual scale of the benefit is a bit disappointing.  With an average 2.2 years of followup for 13784 people in the treatment group, about 200 heart attacks, strokes, or heart disease deaths were postponed or prevented.  At current US prices of $14,000 per year, that would cost over US$420 million.

So, two cheers for genomics in drug development.

March 17, 2017

Is ibuprofen killing you?

The Herald story starts off

Commonly bought over-the-counter painkillers including ibuprofen have been linked to a significant increased risk of cardiac arrest.

The research paper is here (but paywalled).

First, it’s important to remember that “significant” in this context means “detectable” rather than “important.” The risk was higher by about 30%, but cardiac arrest is fairly rare.  With ten years of complete data from Denmark (about 5.5 million people) the researchers accumulated 30,000 cardiac arrests: that’s about five cases per ten thousand people per year.

As usual, this is observational data looking at correlations; the harmful effect, if it’s real, is too small to see reliably in clinical trials.  The researchers used a clever study design where they compared use of painkillers in a cardiac-arrest patient both with the same patient at times in the past and with different patients at the same time.  Differences between people that are constant over time (like smoking) will cancel out of the analysis; differences over time that are constant between people (like season) will also cancel out.  The design doesn’t cancel out non-constant factors like starting an exercise programme that leaves your muscles and joints sore.  It’s not unreasonable that a risk difference this small could be explained by confounding factors.

There’s something more important wrong with the story, though. You might wonder how people who have cardiac arrest get asked about their painkiller use. They didn’t; the study used prescription data.  For many of the painkillers, prescription is the only source; in particular, that’s the case for diclofenac (Voltaren), where the apparent risk increase in the study was a bit larger.

Ibuprofen, however, is available over the counter in Denmark, just as it is here. It’s available in fairly small packages, and is labelled for short-term use, just as it is here. Over-the-counter sale is what the story is basically about, but the study didn’t look at over-the-counter use at all.

March 16, 2017

Don’t say we didn’t warn you

From Stuff

3/2017: “New Zealand homeowners are being told to fix their interest rates now if they want to avoid a looming increase.”

Also from Stuff, all either headline or lead:

12/2016: Mortgage holders urged to fix as US interest rates rise

7/2016: Warning interest rates may not have much further to fall

3/2014: Time to fix loan on your house

1/2014: Rush to fix home loan rates before Reserve Bank acts

9/2013: “Economists say homeowners have officially missed the boat on locking in cheap fixed mortgage rates.”

4/2011: Now’s the time to fix mortgages: Tower

8/2009: Fix your mortgage before rates rise

There are good reasons to believe today’s story, but presumably there were for the past stories, too.

March 13, 2017

But, fear itself

new research paper from Alastair Woodward and co-workers at the University of Auckland looks at the the risks of cycling in New Zealand. Jamie Morton at the Herald has written about it.  Basically, cycling isn’t as dangerous as you probably thought: the risk of an injury severe enough to report to ACC or to go to the emergency department is about one incident per 10,000 half-hour trips.  Or, for me, about once in 25-30 years.

There are two caveats for this as a pro-cycling message.  First, there’s some selection bias: the people who currently cycle are more likely to have safe routes available than those who currently don’t cycle — bike paths really work.  So if more people started cycling with the current infrastructure the ‘safety in numbers’ effect would be reduced by the increased use of dangerous roads.

Second,  it isn’t just actual injury that’s a problem.  The research paper talks about the social context of risk perception, and how the fact that cycling is regarded as weird makes the risks seem higher, which is true and an important factor. But. One morning recently, I stopped at the traffic lights coming off Grafton Bridge, and the bus behind me didn’t.  I didn’t come that close to being hit; It’s still not a fun way to start the day.  Russell Brown, who can actually write, covers this aspect better than I can.  He concludes

Cycling is much safer than people think. But until things change, fear of cycling will keep many reasonable people off the roads.

March 12, 2017

Highchart of the week


It’s not a piechart, because the wedges don’t add up to anything, which is the only possible justification for a pie chart.  On the other hand, unlike the pizzachart it is trying to display numerical data.

Also, “51% of Americans have tried marijuana today” is presumably not the intended reading, but the graphic doesn’t make that as clear as it might.

And the source for the data isn’t a guy named Moe. That’s an abbreviation for Margin of Error.  Google suggests the source is a CBS News Poll (PDF report), but that’s from last year.

(via @seanjtaylor)