Posts written by Thomas Lumley (1690)


Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient

January 25, 2016

Hangover cure?

“The ‘kudzu’ could cure your hangover”, says the Herald (from the Daily Mail)

We’ve had stories based on unpublished research before. This one is a bit special, because the ‘unpublished research’ is a BBC television programme that won’t be shown for a few days (or, at all, in NZ). And because it didn’t study hangovers or people with hangovers.

The BBC programme is apparently going to report a placebo-controlled crossover trial of how much free booze people drank when they had taken kudzu root extract or a dummy pill — they say they saw a 20% difference. The design is a good one, and given the general theme of the show there probably isn’t much risk of publication bias. Having a reasonably good design fits with seeing a benefit that’s only about half of what was seen in earlier US research.

The big problems with the BBC study are that it’s probably small (the stories don’t say) and doesn’t (so far)give any idea of the uncertainty in the 20% reduction, and that open-bar drinking in a television science experiment may be importantly different from the sort of drinking they’d like to decrease.

As science, the BBC study might actually be ok. But reducing alcohol consumption by 20%, if it does, is unlikely to have a big impact in preventing hangovers. And there is not the slightest suggestion in the research that kudzu root would work as a hangover cure.



January 24, 2016

Tracing a data factoid

I saw on Twitter this morning the claim “90% of the worlds data was created in the last 12 months”, raising the twin questions “Says who?” and “What does that even mean?”.

The tweet linked to a story at the Huffington Post, which attributed the claim to Constellation Research, but with no further information on how they found it out or when, or what they meant by it. Further Google searches find that most occurrences of the claim are either completely unsourced, attributed to the guy who wrote the HuffPo story, or also just say “Constellation Research”.

IBM has claimed “Everyday, we create 2.5 quintillion bytes of data–so much that 90% of the data in the world today has been created in the last two years alone.” But that’s two years, not 12 months. And they’ve been saying it for some time:

In 2012, the skeptics site at discussed the IBM claim and found it plausible then — assuming that ‘data has been created’ meant that ‘data has been stored in some permanent medium of the sort IBM sells’. It’s not just business data, as some of the citations imply — it will include cat videos on mobile phones, data from the Large Hadron Collider, the NSA’s archive of your email, and lots and lots of porn.

It still could be that Constellation Research has a new estimate, not the recycled one from the days before Siri. However, a 2015 Oracle presentation (PDF, slide 7) says “90% of the world’s data has been created in the last two years“, and cites “ Constellation Research: “Businesses Must Answer the Call for Cloud Based Integration”“. So I’m guessing maybe not.

January 22, 2016

There’s nothing like a good joke

Q: Did you see that laughing can burn as many calories as a brisk walk?

A: In the Herald? Yes.

Q: Can it?

A: A very short brisk walk, yes.

Q: What do you mean?

A: They say “flat out, uncontrollable, roll on the floor howling – was found to burn 120 calories an hour, similar to the amount burnt by walking at a moderate pace.”

Q: So you just need to do that for an hour, nonstop?

A: That’s right.

Q: This was a British scientist, right?

A: ….yes?

Q: The BBC has pretty good comedies

A: Not that good

Q: Do they have any real estimates of how much exercise you could get?

A: Not in the Herald, but another newspaper says their estimated maximum for the funniest show they looked at was 32 ­calories of laughter over 45 minutes.

Q: So like a quarter of an hour of brisk walking?

A: Pretty much

Q: Which show?

A: Taskmaster

Q: ?

A: I think you had to be there.

Q: And how about humour producing wry smiles and the occasional snigger?

A: That burns about as much energy as reading blogs.

Q: It doesn’t sound like a serious suggestion. Is it possible that the papers which printed this story just didn’t get the joke?

A: Could easily be.

January 21, 2016

Mining uncertainty

The FDA collects data on adverse events in people taking any prescription drugs. This information is, as it should be, available for other uses. I’ve been involved in research using it.

The data are also available for less helpful purposes. As Scott Alexander found,  if you ask Google whether basically anything could cause basically anything, there are companies that make sure Google will return some pages reporting that precise association.  And, as he explains, this is serious.

For example, I tried “Adderall” and “plantar fasciitis” as an implausible combination and got 4 hits based on FDA data. And “Accutane” and “plantar fasciitis”, and “Advair” and “plantar fasciitis”, and “acyclovir” and “plantar fasciitis”. Then I got bored.

It’s presumably true that there are people who have been taking Adderall and at the same time have had plantar fasciitis. But given enough patients to work with, that will be true for any combination of drug and side effect. And, in fact, the websites will happily put up a page saying there are no reported cases, but still saying “you are not alone” and suggesting you join their support group.

These websites are bullshit in the sense of philosopher Harry Frankfurt: it is irrelevant to their purpose whether Adderall really causes plantar fasciitis or not. They make their money from the question, not from the answer.


(via Keith Ng)

January 19, 2016

Rebooting your immune system?

OneNews had a strange-looking story about multiple sclerosis tonight, with lots of footage of one British guy who’d got much better after treatment, and some mentions of an ongoing trial. With the trial still going on, it wasn’t clear why there was publicity now, or why it mostly involved just one patient.

I Google these things so you don’t have to.

So. It turns out there was a new research paper behind the publicity. There is an international trial of immune stem cell transplant for multiple sclerosis, which plans to follow patients for five years after treatment. The research paper describes what happened for the first three years.

As the OneNews story says, there has been a theory for a long time that if you wipe out someone’s immune system and start over again, the new version wouldn’t attack the nervous system and the disease would be cured. The problem was two-fold. First, wiping out someone’s immune system is an extraordinarily drastic treatment — you give a lethal dose of chemotherapy, and then rescue the patient with a transplanted immune system. Second, it didn’t work reliably.

The researcher behind the current trial believes that the treatment would work reliably if it was done earlier — during one of the characteristic remissions in disease progress, rather than after all else fails. This trial involves 25 patients, and so far the results are reasonably positive, but three years is really to soon to tell whether the benefits are worth the treatment. Even with full follow-up of this uncontrolled study it probably won’t be clear exactly who the treatment is worthwhile for.

Why the one British guy? Well,

The BBC’s Panorama programme was given exclusive access to several patients who have undergone the stem cell transplant.

The news story is clipped from a more in-depth current-affairs programme. That BBC link also shows a slightly worrying paranoid attitude from the lead researcher

He said: “There has been resistance to this in the pharma and academic world. This is not a technology you can patent and we have achieved this without industry backing.”

That might explain pharma, but there’s no real reason for the lack of patents to be a problem for academics. It’s more likely that doctors are reluctant to recommend ultra-high-dose chemotherapy without more concrete evidence. After all, it was supposed to work for breast cancer and didn’t, and it was theorised to work for HIV and doesn’t seem to. And at least in the past it didn’t work reliably for multiple sclerosis.

All in all, I think the OneNews story was too one-sided given the interim nature of the data and lack of availability of the treatment.  It could also have said a bit more about how nasty the treatment is.  I can see it being fine as part of a story in a current affairs programme such as Panorama, but as TV news I think it went too far.

January 18, 2016

Supplement pushing

The Herald has a Daily Mail story about vitamin D for making you generally feel better. It’s not so long ago that the NZ media had a lot of less supportive coverage on vitamin D — Ian Reid, Mark Bolland, and Andrew Grey won the Prime Minister’s Science Prize last year for their work showing that calcium and vitamin D aren’t all they’re cracked up to be.

The story does have some new evidence.

In the study, by a medical team in Edinburgh, volunteers were asked to cycle for 20 minutes. They were then given either a placebo or vitamin D and, two weeks later, were asked to cycle for 20 minutes again.

The buck needs to stop somewhere

From Vox:

Academic press offices are known to overhype their own research. But the University of Maryland recently took this to appalling new heights — trumpeting an incredibly shoddy study on chocolate milk and concussions that happened to benefit a corporate partner.

Press offices get targeted when this sort of thing happens because they are a necessary link in the chain of hype.  On the other hand, unlike journalists and researchers, their job description doesn’t involve being skeptical about research.

For those who haven’t kept up with the story: the research is looking at chocolate milk produced by a sponsor of the study, compared to other sports drinks. The press release is based on preliminary unpublished data. The drink is fat-free, but contains as much sugar as Coca-Cola. And the press release also says

“There is nothing more important than protecting our student-athletes,” said Clayton Wilcox, superintendent of Washington County Public Schools. “Now that we understand the findings of this study, we are determined to provide Fifth Quarter Fresh to all of our athletes.”

which seems to have got ahead of the evidence rather.

This is exactly the sort of story that’s very unlikely to be the press office’s fault. Either the researchers or someone in management at the university must have decided to put out a press release on preliminary data and to push the product to the local school district. Presumably it was the same people who decided to do a press release on preliminary data from an earlier study in May — data that are still unpublished.

In this example the journalists have done fairly well: Google News shows that coverage of the chocolate milk brand is almost entirely negative.  More generally, though, there’s the problem that academics aren’t always responsible for how their research is spun, and as a result they always have an excuse.

A step in the right direction would be to have all research press releases explicitly endorsed by someone. If that person is a responsible member of the research team, you know who to blame. If it’s just a publicist, well, that tells you something too.

January 17, 2016

Not science yet

I’ve written before about the problem of unpublished science in the news: the news story won’t (can’t) give much detail, and there’s no way to find it out.  Stuff has gone one step further:

Recent animal studies show sleep’s cleansing process in action. But now scientists at Oregon Health & Science University are preparing to conduct a study on humans that would further explain deep sleep’s effect on human brains.

The ‘animal studies’ claim comes without any source. Fortunately the Google comes to the rescue and suggests it’s the research in this story from 2013. Interestingly, a couple of subsequent mouse studies have also found brain problems from interrupted sleep — but since each of the three studies found a different problem, with no overlap, this isn’t as supportive as it sounds.

It would matter less if it weren’t for the first sentence of the story

Forget about needing beauty sleep. It’s your brain that may suffer the most from a lack of deep shut eye.

There’s a definite suggestion that this is a risk factor you can do something about, especially as this is in the “Well & Good” section of the Stuff site.

Even if the Oregon research had been carried out and published, it might well not justify that sort of implication. Research that they’re still preparing to do certainly doesn’t.


[Update: It’s getting to be a trend: the Herald also has a story about research that hasn’t happened yet, on addiction.]

January 15, 2016

When you don’t find any

The Icelandic Ethical Humanist Association commissioned a survey on religion. For people who don’t want to read the survey report (in PDF, in Icelandic), there’s a story at Iceland Magazine. The main point is in the headline: 0.0% of Icelanders 25 years or younger believe God created the world, new poll reveals.

That’s a pretty strong claim, so what did the survey actually do? Well, here you do need to read the survey report (or at least feed snippets of it to Google Translate). Of the people they sampled, 109 were in the lowest age category, which is ‘younger than 25’.  None of the 109  reported believing “God created the world” vs “The world was created in the Big Bang”.

Now, that’s not a completely clean pair of alternatives, since a fair number of people — the Pope, for example — say they believe both, but it’s still informative to some extent. So what can we say about sampling uncertainty?

A handy trick for situations like this one is the ‘rule of 3’.  If you ask N people and none of them is a creationist, a 95% confidence upper bound for the population proportion is 3/N. So, “fewer than 3% of Icelanders under 25 believe God created the world”

Who got the numbers, how, and why?

The Dominion Post has what I’m told is a front page story about school costs, with some numbers:

For children starting state school this year, the total cost, including fees, extracurricular activities, other necessities, transport and computers, by the time they finish year 13 in 2028 is estimated at $35,064 by education-focused savings trust Australian Scholarship Group.

That increases to $95,918 for a child at a state-integrated school, and $279,807 for private school.

Given that the figures involve extrapolation of both real cost increases and inflation thirteen years into the future, I’m not convinced that a whole-education total is all that useful. I would have thought estimates for a single year would be more easily interpreted.  However, that’s not the main issue.

ASG do this routinely. They don’t have the 2016 numbers on their website yet, but they do have last year’s version. Important things to note about the numbers, from that link:

ASG conducted an online education costs survey among its members during October 2013. The surveys covered primary and secondary school. In all, ASG received more than 1000 survey responses.

So, it’s a non-random, unweighted survey, probably with a low response rate, among people signed up for an education-savings programme. You’d expect it to overestimate, but it’s not clear how much. Also

Figures have been rounded and represent the upper ranges that parents can reasonably expect to pay

‘Rounded’ is good, even though they don’t actually show much sign of having been rounded. ‘Represent the upper ranges’ is a bit more worrying when there’s no indication of how this was done — and when the Dom Post didn’t include this caveat in their story.