Posts filed under Experiments (16)

Despite the date, this is not in any way an April Fools post

• “Data is not killing creativity, it’s just changing how we tell stories”, from Techcrunch
• Turning free-form text into journalism: Jacob Harris writes about an investigation into food recalls (nested HTML tables are not an open data format either)
• Green labels look healthier than red labels, from the Washington Post. When I see this sort of research I imagine the marketing experts thinking “how cute, they figured that one out after only four years”
• Frances Woolley debunks the recent stories about how Facebook likes reveal your sexual orientation (with comments from me).  It’s amazing how little you get from the quoted 88% accuracy, even if you pretend the input data are meaningful.  There are some measures of accuracy that you shouldn’t be allowed to use in press releases.

• A post at Scientific American about covering clinical trials, for journalists and readers.  It’s a summary from the Association of Health Care Journalists annual conference. Starts out “My message: Ask the hard questions.”
• Asking the hard questions is also useful in covering surveys.  Stuff reports “Kiwi leaders amongst the world’s riskiest”,

New Zealand leaders are among the most likely in the world to ignore data and fail to seek a range of opinions when making decisions

with no provenance except that this was based on a 600,000 person survey of managers and professionals by SHL.  Before trying to track down any more detail, just think: how could this have worked? How would you get reliable information to support those conclusions from each of 600,000 people? 

• You may have heard about the famous Hawthorne experiment, where raising light levels in a factory improved output, as did lowering them, as did anything else experimental. The original data have been found and this turns out not to be the case.

There has been (justified) wailing and gnashing of teeth over recent year-9 maths comparisons, and the Herald reports that a `back to basics’ system is being considered

Auckland educator Des Rainey, who did the research with teachers to test his home-made Kiwi Maths memorisation system, said the results came as a shock to the teachers and made him doubt his programme could work.

But after a year of practising multiplication and division on the Kiwi Maths grids for up to 10 minutes a day, the students more than doubled their speed.

This program looks promising, but why is anyone even talking about implementing a major nationwide intervention based on a small, uncontrolled before/after comparison measuring a surrogate outcome?

That is, unless you believe teachers and schoolchildren are much less individually variable than, say, pneumococci, you would want a randomised controlled comparison, and since presumably Des Rainey would agree that speed of basic arithmetic is important primarily because it’s a foundation for actual numeracy, you’d want to measure the success of the program based on numeracy tasks rather than on arithmetic speed. The results being reported are what the medical research community would call a non-randomised Phase IIa efficacy trial — an important stepping stone, but not a basis for policy.

Of course, that’s not how education works, is it?

The Herald has a story claiming that facial proportions indicate racism (in men).  Well, they have a headline claiming that. The story (and the research paper, even more explicitly) pretty much contradicts the headline, and says that facial proportions have nothing to do with racism but indicate whether men write magazine articles about express their racist views or hide them.

If you believe the story, the relationship is very strong

Looking at the photos from the first study, a new group of participants evaluated men with wider, shorter faces as more prejudiced, and they were able to accurately estimate the target’s self-reported prejudicial beliefs just by looking at an image of his face.

and to be fair to the journalist, that’s what the researchers said.  If you look at their actual results, it’s not what they found.

They found an average difference of 1.92 on a 6-point perceived-racism scale for men who differ by 1 unit on the facial proportion scale.  The full range of the facial proportion scale appears to only be about 0.7 units. The paper doesn’t tell us the actual distribution of the measurements, but according to another research paper I found on the internets, the standard deviation of this facial proportion scale is about 0.12.  That means two randomly chosen men would differ by about 0.17 units, and the relationship  would predict a difference in the 6-point perceived-racism scale of about 0.3 units.  The association with self-reported racism was about as strong, though I haven’t been able to find enough information to compute the predicted differences (it shouldn’t be this hard).

In my book, that’s not an “accurate estimate”.

Stuff has a story from the Sydney Morning Herald on clinical trials in India.  The basic claims are damning if true:

…clinical drug trials are at the centre of a growing controversy in India, as evidence emerges before courts and, in government inquiries, of patients being put onto drug trials without their knowledge or consent…

With a very few exceptions (eg some trials of emergency resuscitation techniques and some minimal-risk cluster-randomised trials of treatment delivery)  it is absolutely fundamental that trial participants give informed consent. Trial protocols are supposed to be reviewed in advance to make sure that participants aren’t asked to consent to unreasonably things, but consent is still primary.  This isn’t just a technical detail, since researchers who were unclear on the importance of consent have often been bad at other aspects of research or patient care.

The Fairfax story mixes in the claimed lack of consent with other claims that are either less serious or not explained clearly. For example

Figures from the drugs controller- general show that in 2011 there were deaths during clinical trials conducted by, or on behalf of, Novartis, Quintiles, Pfizer, Bayer, Bristol Mayer Squibb, and MSD Pharmaceutical.

Of course there were deaths in clinical trials. If you are comparing two treatments for a serious illness, the trial participants will be seriously ill.  When you need to know if a new treatment reduces the risk of death, the only way to tell is to do a study large enough that some people are expected to die.  Even if improved survival isn’t directly what you’re measuring, a large trial will include people who die. In the main Women’s Health Initiative hormone replacement trial, for example, 449 women had died by the time the trial was stopped.  The question isn’t whether there were deaths, it’s whether there were deaths that wouldn’t have occurred if the trials had been done right.

There’s also a claim that families of participants who died were not given adequate compensation as part of the trial.  If there had been consent, this wouldn’t necessarily matter. Lots of trials in developed countries don’t specifically compensate participants or relatives, and there’s actually some suspicion of those that do, because it provides another incentive to participate even if you don’t really want to.

Other sources: Times of India, Chemistry World, a couple of review articles, the Nuremberg Code

An article in Nature (not free access, unfortunately) by Australian molecular biologist David L. Vaux

 ”Experimental biologists, their reviewers and their publishers must grasp basic statistics, or sloppy science will continue to grow.”

This doesn’t come as a surprise to statisticians, but it is nice to get the support from the biology side.  His recommendations are also familiar and welcome

How can the understanding and use of elementary statistics be improved? Young researchers need to be taught the practicalities of using statistics at the point at which they obtain the results of their very first experiments.

[Journals] should refuse to publish papers that contain fundamental errors, and readily publish corrections for published papers that fall short. This requires engaging reviewers who are statistically literate and editors who can verify the process. Numerical data should be made available either as part of the paper or as linked, computer-interpretable files so that readers can perform or confirm statistical analyses themselves.

Professor Vaux goes on to say

When William Strunk Jr, a professor of English, was faced with a flood of errors in spelling, grammar and English usage, he wrote a short, practical guide that became The Elements of Style(also known as Strunk and White). Perhaps experimental biologists need a similar booklet on statistics.

And here I have to quibble. Experimental biologists already have too many guides like Strunk & White, full of outdated prejudices and policies that the authors themselves would not follow.  What we need is a guide that lays out how good scientists and statisticians actually do handle common types of experiment (ie, evidence-based standard recipes), together with some education on the basic principles: contrasts, blocking, randomization, sources of variation, descriptions of uncertainty. And perhaps a few entertaining horror stories of Doing It Rong and the consequences.

Philosophers have argued about whether it’s even conceivable to have pain without the physical sensation. According to 3News (and other media outlets worldwide), University of Chicago neuroscientists don’t have a problem with this:

Mathematics can be difficult, and a new study shows even thinking about doing it can physically hurt.

Of course, that’s not quite what the study found (and credit to 3News for linking), though it does seem to be what the researchers said they found. The study was in people with ‘high levels of math anxiety’ and the abstract says

We show that, when anticipating an upcoming math-task, the higher one’s math anxiety, the more one increases activity in regions associated with visceral threat detection, and often the experience of pain itself (bilateral dorso-posterior insula).

That is, some of the parts of the brain that are active during pain or threat were also active when anticipating a maths task, even though there was no actual pain reported.

A simpler explanation might be that if you’re scared of maths, then your brain looks as if you’re scared of something.  Although the researchers don’t believe this, they do actually concede it is an alternative explanation in the discussion section of the paper

the INSp activity we found could be reflective of something else. For example, it has been suggested that INSp activity is not so much reflective of nocioception, but rather reflects detection of events that are salient for (e.g., threatening to) bodily integrity, regardless of the input sensory modality

(via)

Q:  Have you seen the headline: “Skipping breakfast makes you gain weight: study”?

A:  If that’s the one with the chocolate cupcake photo, yes.

Q:  Was this just another mouse study, or did they look at weight gain in people?

A: People, yes, but they didn’t measure weight gain.

Q: But doesn’t the headline say “makes you gain weight”?

A: Indeed.

Q: So what did they do?

A: They measured brain waves, and how much pasta lunch people ate. The people who skipped breakfast ate more.

Q: So it was a lab experiment.

A: You can’t really tell from the Herald story, which makes it sound as though the participants just chose whether or not to have breakfast, but yes.  If you look at the BBC version, it says that the same people were measured twice, once when they had breakfast and once when they didn’t.

Q: And how much more lunch did they eat when they didn’t eat breakfast?

A: An average of 250 calories more.

Q: How does that compare to how much they would have eaten at breakfast?

A:  There were brain waves, as well.

Q: How many calories would the participants have eaten at breakfast?

A: The part of the brain thought to be involved in “food appeal”, the orbitofrontal cortex, became more active on an empty stomach.

Q: Are you avoiding the question about breakfast?

A: Why would you think that?  The breakfast was 730 calories.  But the MRI imaging showed that fasting made people hungrier

Q: Isn’t 730 more than 250?

A: Comments like that are why people hate statisticians.

Stuff is reporting ”Food and learning connection shot down”,based on a local study

Researchers at Auckland University’s School of Population Health studied 423 children at decile one to four schools in Auckland, Waikato and Wellington for the 2010 school year.

They were given a free daily breakfast – Weet-Bix, bread with honey, jam or Marmite, and Milo – by either the Red Cross or a private sector provider.

My first reaction on reading this was: why didn’t they take this opportunity to do a randomised trial, so we could actually get reliable data.  So I went to the Cochrane Library to see what randomised trials had been done in the past. These have mostly been in developing countries and have found improvements in growth, but smaller differences in school performance.

Then I tried asking the Google, and its second link was a paper by Dr Ni Mhurchu, the researcher mentioned in the story, detailing the plans for a randomised trial of school breakfasts in Auckland.  At that point it was easy to find the results, and see that in fact Stuff is talking about a randomized trial. They just didn’t think it was important enough to mention that detail.

To the extent that one can trust the Stuff story at this point, there seem to be three reactions:

• I don’t believe it because my opinions are more reliable than this research
• Lunch would work even if breakfast didn’t
•  We should be making sure kids have breakfast even if it doesn’t improve school performance.

The latter two responses are perfectly reasonable positions to take (though they’re more convincing where they were taken before the results came out).  School lunches might be more effective than breakfasts, and the US (hardly a hotbed of socialism) has had a huge school nutrition program for 60 years.

Still, if we’re going to supply subsidised meals to school kids, we do need to know why we’re doing it and what we expect to gain.    This study is one of the first to go beyond just saying that the benefits are obvious.

Mind the Science Gap is a blog from the University of Michigan:

Each semester, ten Master of Public Health students from the University of Michigan participate in a course on Communicating Science through Social Media. Each student on the course is required to post weekly articles here as they learn how to translate complex science into something a broad audience can understand and appreciate. And in doing so they are evaluated in the most brutal way possible – by you: the audience they are writing for!

The post that attracted me to the blog was on sugar and hyperactivity in kids, not just for the science, but because someone has actually found a good use for animated GIFs in communicating information: click to see the effect, since embedding it in WordPress seems to kill it.