Posts filed under Evidence (34)

The Herald has a completely over-the-top presentation of what might be an important issue. The headline is “Hospital choice key to kids’ survival”, and the story starts off

Where ambulances take badly injured children first seems to affect their chances, paediatric surgeons say.

Starship children’s hospital surgeons have found that sending badly injured children to the wrong hospital may be contributing to a child death rate from injuries that is twice the rate of Australia’s.

The data:

Six (7 per cent) of the 88 children who went first to Middlemore died, but so did one (8 per cent) of the 12 who went directly to Starship.

That is, to the extent the data tell us anything, the evidence is against the headline.  Of course, the uncertainties are huge: a 95% confidence interval for the relative odds of dying after being sent to Middlemore goes from a 40-fold decrease to a 12-fold increase.  There’s basically no information in the survival data.

So, how much of the two-fold higher rate of death in NZ compared to Australia could reasonably be explained by suboptimal hospital choice? One of the surgeons involved in the study says

… overseas research showed that a good trauma protocol system could cut the death rate for injured adults by 20 to 30 per cent, but there was no good data for children.

That is, hardly any of the difference between NZ and Australia — especially as this specific hospital-choice issue only applies to one sector of one city in New Zealand, with less than 10% of the national population.

On the other hand, we see

The head of Starship’s emergency department, Dr Mike Shepherd, said the major factors contributing to New Zealand’s high fatal injury rate for children lay outside the hospital system in policies such as driver blood-alcohol limits, graduated driver licensing, and laws requiring children’s booster seats and swimming pool fences.

That sounds plausible, but if it’s the whole story you would expect high levels of non-fatal as well as fatal injuries. The overall rate of hospitalisations for injuries in children 0-14 years is almost identical in NZ (1395 per 100 000 per year, p29) and Australia (‘about’ 1500 per 100 000 per year, page v).

A long piece in the Washington Post: by Ezra Klein, recommended by Atul Gawande

Brenner puts it more vividly. “There is a bias in medicine against talking to people and for cutting, scanning and chopping into them. If this was a pill or or a machine with these results it would be front-page news in the Wall Street Journal. If we could get these results for your grandmother, you’d say, ‘Of course I want that.’ But then you’d say, what are the risks? Does she need to have chemotherapy? Does she need to be put in a scanner? Is it a surgery? And you’d say, no, you just have to have a nurse come visit her every week.”

• BBC news report on the Amanda Knox case and misuse of statistics: judge did not believe that repeating a test could increase its accuracy (via Mark Wilson)
• A tool for finding stories based on light makeover of press releases. Unfortunately, it often doesn’t detect recycling of stories from sites such as Medical Daily and Science Daily, which do enough rewriting to mask the sources.
• New York Times opinion piece on evidence and science reporting — yes, distinguish experimental and observational studies, but also distinguish small exploratory studies from larger confirmatory ones. (via @brettkeller)
• Using anecdotes rather than data to convince patients. On one hand, speaking to people in language they understand is good; on the other hand, you can use anecdotes to support anything.

I don’t usually bother with general nutrition stories that don’t contain any direct reference to research, but the Herald story about berries was irresistible. There are lots of biologically active compounds in berries, and many of them have been shown to have interesting properties in test-tubes or mice. As you know by now,  this sort of interesting biochemistry is important because it occasionally translates to genuine health benefits, so you should be asking what the human clinical research shows.

If you go to the Cochrane Library (which is free to everyone in New Zealand), and look for clinical research in humans involving blueberries or cranberries you don’t find much. The only topic with enough information to draw any sort of conclusion is on cranberry juice to prevent urinary tract infections. Which it basically doesn’t. The plain-language summary says

Despite the date, this is not in any way an April Fools post

• “Data is not killing creativity, it’s just changing how we tell stories”, from Techcrunch
• Turning free-form text into journalism: Jacob Harris writes about an investigation into food recalls (nested HTML tables are not an open data format either)
• Green labels look healthier than red labels, from the Washington Post. When I see this sort of research I imagine the marketing experts thinking “how cute, they figured that one out after only four years”
• Frances Woolley debunks the recent stories about how Facebook likes reveal your sexual orientation (with comments from me).  It’s amazing how little you get from the quoted 88% accuracy, even if you pretend the input data are meaningful.  There are some measures of accuracy that you shouldn’t be allowed to use in press releases.

There’s a new UK report by Ben Goldacre, “Building Evidence into Education”, which has been welcomed by the Teacher Development Trust

Part of the introduction is worth quoting in detail:

Before we get that far, though, there is a caveat: I’m a doctor. I know that outsiders often try to tell teachers what they should do, and I’m aware this often ends badly. Because of that, there are two things we should be clear on.

Firstly, evidence based practice isn’t about telling teachers what to do: in fact, quite the opposite. This is about empowering teachers, and setting a profession free from governments, ministers and civil servants who are often overly keen on sending out edicts, insisting that their new idea is the best in town. Nobody in government would tell a doctor what to prescribe, but we all expect doctors to be able to make informed decisions about which treatment is best, using the best currently available evidence. I think teachers could one day be in the same position.

Secondly, doctors didn’t invent evidence based medicine. In fact, quite the opposite is true: just a few decades ago, best medical practice was driven by things like eminence, charisma, and personal experience. We needed the help of statisticians, epidemiologists, information librarians, and experts in trial design to move forwards. Many doctors – especially the most senior ones – fought hard against this, regarding “evidence based medicine” as a challenge to their authority.

In retrospect, we’ve seen that these doctors were wrong. The opportunity to make informed decisions about what works best, using good quality evidence, represents a truer form of professional independence than any senior figure barking out their opinions. A coherent set of systems for evidence based practice listens to people on the front line, to find out where the uncertainties are, and decide which ideas are worth testing. Lastly, crucially, individual judgement isn’t undermined by evidence: if anything, informed judgement is back in the foreground, and hugely improved.

This is the opportunity that I think teachers might want to take up.

There has been (justified) wailing and gnashing of teeth over recent year-9 maths comparisons, and the Herald reports that a `back to basics’ system is being considered

Auckland educator Des Rainey, who did the research with teachers to test his home-made Kiwi Maths memorisation system, said the results came as a shock to the teachers and made him doubt his programme could work.

But after a year of practising multiplication and division on the Kiwi Maths grids for up to 10 minutes a day, the students more than doubled their speed.

This program looks promising, but why is anyone even talking about implementing a major nationwide intervention based on a small, uncontrolled before/after comparison measuring a surrogate outcome?

That is, unless you believe teachers and schoolchildren are much less individually variable than, say, pneumococci, you would want a randomised controlled comparison, and since presumably Des Rainey would agree that speed of basic arithmetic is important primarily because it’s a foundation for actual numeracy, you’d want to measure the success of the program based on numeracy tasks rather than on arithmetic speed. The results being reported are what the medical research community would call a non-randomised Phase IIa efficacy trial — an important stepping stone, but not a basis for policy.

Of course, that’s not how education works, is it?

There are new reports, according to the Herald, that synthetic cannabinoids are ‘associated’ with suicidal tendencies in long-term users.  One difficulty in evaluating this sort of data is the huge peak in suicide rates in young men.  Almost anything you can think of that might be a bad idea is more commonly done by young men than by other people, so an apparent association isn’t all that surprising.  There is also the problem with direction of causation — the sorts of problems that make suicide a risk might also increase drug use — and difficulties even in getting a reasonable estimate of the denominator, the number of people using the drug. Serious, rare effects of a recreational drug are the hardest to be sure about, and the same is true of prescription medications.  It took big randomized trials to find out that Vioxx more than doubled your rate of heart attack , and a study of 1500 lung-cancer cases even to find the 20-fold increase in risk from smoking.

In this particular example there is additional supporting evidence. A few years back there was a lot of research into anti-cannabinoid drugs for weight loss (anti-munchies), and one of the things that sank these was an increase in suicidal thoughts in the patients in the early randomized trials.  It’s quite plausible that the same effect would happen as a dose of the cannabinoid wears off.

In general, though, this is the sort of effect that the proposed testing scheme for psychoactive drugs will have difficulty finding, or ruling out.

David Farrar (among others) has written about a recent Coroner’s recommendation that high-visibility clothing should be compulsory for cyclists.  As he notes, “ if you are cycling at night you are a special sort of moron if you do not wear hi-vis gear”, but he rightly points out that isn’t the whole issue.

It’s easy to analyse a proposed law as if the only changes that result are those the law intends: everyone will cycle the same way, but they will all be wearing lurid chartreuse studded with flashing lights and will live happily ever after.  But safety laws, like other public-health interventions, need to be assessed on what will actually happen.

Bicycle helmet laws are a standard example.  There is overwhelming evidence that wearing a bicycle helmet reduces the risk of brain injury, but there’s also pretty good evidence that requiring bicycle helmets reduces cycling. Reducing the number of cyclists is bad from an individual-health point of view and also makes cycling less safe for those who remain. It’s not obvious how to optimise this tradeoff, but my guess based on no evidence is that pro-helmet propaganda might be better than helmet laws.

Another example was a proposal by some US airlines to require small children to have their own seat rather than flying in a parent’s lap. It’s clear that having their own seat is safer, but also much more expensive.  If any noticeable fraction of these families ended up driving rather than flying because of the extra cost, the extra deaths on the road would far outweigh those saved in the air.

It’s hard to predict the exact side-effects of a law, but that doesn’t mean they can be ignored any more than the exact side-effects of new medications can be ignored. The problem is that no-one will admit they don’t know the effects of a proposed law.  It took us decades to persuade physicians that they don’t magically know the effects of new treatments; let’s hope it doesn’t take much longer in the policy world.

[PS: yes, I do wear a helmet when cycling, except in the Netherlands, where bikes rule]

This weekend, Christchurch had its biggest aftershock for six months.  The moon was substantially further from the earth than average.