How can a sham medical procedure provide huge benefits? And why do we still do them in a world of randomised, blinded trials? Thomas Lumley explores the issue in his latest New Zealand Listener column. Click here.
Posts filed under Design of experiments (17)
Every year, the Department of Statistics at the University of Auckland offers summer scholarships to a number of students so they can work with our staff on real-world projects. We’ll be profiling the 2013-2014 summer scholars on Stats Chat. Tania is working with Dr Stephanie Budgett on a project titled First-time mums: Can we make a difference?
“This project is based on the ongoing levator ani study (LA, commonly known as the pelvic floor muscles) from the Pelvic Floor Research Group at the Auckland Bioengineering Institute (ABI), which looks at how the pelvic floor muscles change after first-time mums give birth.
“The aim is to see whether age, ethnicity, delivery conditions and other related factors are associated with the tearing of the muscle. Interestingly, the stiffness of the muscle at rest has been identified as a key factor and is being measured by a specially designed device, an elastometer, that was built by engineers at the ABI.
“Pelvic-floor muscle injury following a vaginal delivery can increase the risks for prolapse where pelvic organs, such as the uterus, small bowl, bladder and rectum, descend and herniate. Furthermore, the muscle trauma may also promote or intensify urinary and/or bowel incontinence.
“Not only do these pelvic- floor disorders cause discomfort and distress, and reduce the mother’s quality of life, and, if left untreated, may lead to major health concerns later in life. Therefore, a statistical model based on key factors elucidated from the study may aid health professionals in deciding the best strategy for delivering a woman’s baby and whether certain interventions are needed.
“I have recently completed my third year of a Bachelor of Science majoring in Statistics and Pharmacology and intend to pursue postgraduate studies. I hope to integrate my knowledge of medical sciences and statistics and specialise in medical statistics.
“Statistics appeals to me because it is a useful field with direct practical applications in almost every industry. I had initially taken the stage one paper as a standalone in order to broaden my knowledge, but eventually realised that I really liked the subject and that it could complement whichever career I have. That’s when I decided to major in statistics, and I’m very glad that I did.
“Over this summer, aside from the project, I am hoping to spend more time with friends and family – especially with my new baby brother! I am also looking forward to visiting the South Island during the Christmas break.”
Sometimes a scientific claim is obviously unreasonable, like when a physicist tells you “No, really, the same electron goes through both slots in this barrier”. You’re all “Wut? No. Can’t be.” They show you the interference pattern. “But did you think of…?” “Yes”. “Couldn’t it be..” “No, we tried that.” “But…” “And that.” “Still, what about…?” “That too.” Eventually you give up and accept that the universe is weird. An electron really can go through two holes at once.
On the other hand, sometimes the claim isn’t backed up that well, like when Stuff tells us “Cookies as addictive as cocaine”. For example, while some rats were given Oreo cookies and others were given cocaine, there weren’t any rates who were offered both, so there wasn’t any direct evaluation of preference, let alone of addiction. The cookies weren’t even compared to the same control as the cocaine — cookies were compared to rice cakes, and cocaine-laced water to plain water.
There’s a more detailed take-down on the Guardian site, by an addiction researcher.
From the Journal of Nutritional Science
Young adult males (n 35) were supplemented with either half or two kiwifruit/d for 6 weeks. Profile of Mood States questionnaires were completed at baseline and following the intervention. No effect on overall mood was observed in the half a kiwifruit/d group; however, a 35 % (P = 0·06) trend towards a decrease in total mood disturbance and a 32 % (P = 0·063) trend towards a decrease in depression were observed in the two kiwifruit/d group. Subgroup analysis indicated that participants with higher baseline mood disturbance exhibited a significant 38 % (P = 0·029) decrease in total mood disturbance, as well as a 38 % (P = 0·048) decrease in fatigue, 31 % (P = 0·024) increase in vigour and a 34 % (P = 0·075) trend towards a decrease in depression, following supplementation with two kiwifruit/d. There was no effect of two kiwifruit/d on the mood scores of participants with lower baseline mood disturbance
From the Otago press release
Eating two kiwifruit a day can improve a person’s mood and give them extra energy, new research from the University of Otago, Christchurch (UOC) shows.
Over a six-week period, normally-healthy young men either ate two kiwifruit a day or half a kiwifruit daily as part of a research study into the potential mood-enhancing effects of the fruit.
Researchers found those eating two kiwifruit daily experienced significantly less fatigue and depression than the other group. They also felt they had more energy. These changes appeared to be related to the optimising of vitamin C intake with the two kiwifruit dose
Eating two kiwifruit a day can improve mood and energy levels, a new University of Otago study shows.
Those eating two kiwifruit were found to experience significantly less fatigue and depression than the others. They also felt they had more energy.
I’m not criticizing the research, which was a perfectly reasonable designed experiment, but if the findings are newsworthy, they are also worth presenting accurately.
1. Walnuts: These nuts look just like a brain, so it makes sense that they’re packed with good stuff for your grey matter.The British Journal of Nutrition reported that eating half a cup of walnuts a day for eight weeks increased reasoning skills by nearly 12 per cent in students.
There’s no way that the appearance of a natural food could possibly be a guide to its nutritional value — how would the walnut know that it’s good for human brains, and why would it care? Pecans, which look a bit like brains, don’t contain the levels of n-3 fatty acids that are supposed to be the beneficial component of walnuts, and fish and flax seeds, which do contain n-3 fatty acids, don’t look like brains.
The story gets two cheers for almost providing a reference: searching on “British Journal of Nutrition walnuts reasoning skills” leads to the paper. It’s a reasonable placebo-controlled randomised experiment, with participants eating banana bread with or without walnuts. The main problem is that the researchers tested 34 measurements of cognitive function or mood, and found a difference in just one of them. As they admit
The authors are unable to explain why inference alone was affected by consumption of walnuts and not the other ‘critical thinking’ subtests – recognition of assumption, deduction, interpretation, and evaluation of arguments.
The prior research summarised in the paper shows the same problem, eg, one dose of walnuts improved one coordination test in rats, but a higher dose improved a different test, and the highest dose didn’t improve anything.
We keep going on here about the importance of publishing clinical trials. Today (in Britain), the BBC program Panorama is showing a documentary about a doctor who has been running clinical trials of the same basic treatment regimen for twenty years, without publishing any results. And it’s not that these are trials that take a long time to run — the participants have advanced cancer. If the treatment was effective, it would have been easy to gather and publish convincing evidence by now, many times over.
These haven’t been especially good clinical trials by usual standards — not randomized, not controlled — and they have been anomalous in other ways as well. For example, patients participating in the trial are charged large sums of money for the treatment being tested (not just for other care), which is very unusual. Unusual, but not illegal. Without published evidence that the treatment works, it couldn’t be sold outside trials, but it’s still entirely legal to charge money for the treatment in research. It’s a bit like whaling.
According to the BBC, Dr Burzynski says it’s not his decision to keep the results secret
He said the medical authorities in the US would not let him release this information: “Clinical trials, phase two clinical trials, were completed just a few months ago. I cannot release this information to you at this moment.”
If true, that would be very unusual. I don’t know of any occasion when the FDA has restricted scientific publication of trial results, and it’s entirely routine to publish results for treatments that have not been approved or even where other research is still ongoing. The BBC also checked with the FDA:
But the FDA told us this was not true and he was allowed to share the results of his trials.
This is all a long way away from New Zealand, and we can’t even watch the documentary, so why am I mentioning it? Last year, the parents of an NZ kid were trying to raise money to send him to the Burzynski clinic, with the help of the Herald. You can’t fault the parents for trying to buy hope at any cost, but you sure can fault the people selling it.
Wikipedia has pretty good coverage if you want more detail.
From the Wellcome Trust Monitor, a survey examining knowledge and attitudes related to biomedical science in the UK
The survey found a high level of interest in medical research among the public – more than seven in ten adults (75 per cent) and nearly six out of ten of young people (58 per cent). Despite this, understanding of how research is conducted is not deep – and levels of understanding have fallen since 2009. While most adults (67 per cent) and half of all young people (50 per cent) recognise the concept of a controlled experiment in science, most cannot articulate why this process is effective.
Two-thirds of the adults that were questioned trusted medical practitioners and university scientists to give them accurate information about medical research. This fell to just over one in ten (12 per cent) for government departments and ministers. Journalists scored lowest on trustworthiness — only 8 per cent of adults trusted them to give accurate information about medical research, although this was an improvement on the 2009 figure of 4 per cent.
Upon receiving the petition the Office of the Clerk undertook a counting and sampling process. Once the signatures had been counted, a sample of signatures was taken using a methodology provided by the Government Statistician.
It’s a good question and I’d already thought of writing about it, so the commenter is getting a temporary reprieve from banishment for not providing a full name. I don’t know for certain, and the details don’t seem to have been published, which is a pity — they would be interesting and educationally useful, and there doesn’t seem to be any need for confidentiality.
While I can’t be certain, I think it’s very likely that the Government Statistician provided the estimation methodology from Statistics New Zealand Working Paper No 10-04, which reviews and extends earlier research on petition counting.
There are several issues that need to be considered
- removing signatures that don’t come with the required information
- estimating the number of eligible vs ineligible signatures
- estimating the number of duplicates
- estimating the margin of error in the estimate
- deciding what level of uncertainty is acceptable
The signatures without the required information are removed completely; that’s not based on sampling. Estimating eligible vs ineligible signatures is fairly easy by checking a sufficiently-large random sample — in fact, they use a systematic sample, taking names at regular intervals through the petition list, which tends to give more precise results and to be more auditable.
Estimating unique signatures is tricky, because if you halve your sample size, you expect to see 1/4 as many duplicates, 1/8 as many triplicates, and so on. The key part of the working paper shows how to scale up the the sample data on eligible, ineligible, and duplicate, triplicate, etc, signatures to get the unique unbiased estimator of the number of valid signatures and its variance.
Once the level of uncertainty is specified, the formulas tell you what sample size to verify and what to do with the results. I don’t know how the sample size is chosen, but it wouldn’t take a very large sample to get the uncertainty down to a few thousand, which would be good enough. In fact, since the methodology is public and the parties have access to the electoral roll in electronic form, it’s a bit surprising that the petition organisers didn’t run a quick check themselves before submitting it.
While distracted by a conference in late January, I missed the next exciting installment of the Edinburgh pomegranate saga.
As you will recall, a research group in Edinburgh have put out press releases in recent years about the impact of pomegranate juice on blood pressure, cortisol (a stress hormone), and testosterone. They haven’t published any scientific papers about these findings, though they have produced a presentation at a scientific conference.
The most recent installment claims that pomegranate extract reduces hunger and food consumption. This study seems to be better designed than the previous ones: participants were randomised to pomegranate extract tablets or placebo for three weeks. They were then given a glass of pomegranate juice and a meal. Those who had been taking the pomegranate extract reported feeling less hungry and ate less — 22% less. It’s a pity the study didn’t measure weight, because if the 22% reduction in food consumption generalised beyond the one experimental meal it would have led to measurable weight loss over three weeks.
Again, this was a premature and unsubstantiated press release, and the experiment has not been published in a peer-reviewed journal, although the researchers do at least say they will be presenting it at a conference later in the year.