Stats Chat

The New York Times has a detailed story about one of the triggers for the Alltrials campaign, the missing studies of Tamiflu

He was curious about one of the main studies on which Dr. Jefferson had relied in his previous analysis. Called the Kaiser study, it pooled the results of 10 clinical trials. But Dr. Hayashi noticed that the results of only two of those trials had been fully published in medical journals. Given that details of eight trials were unknown, how could the researchers be certain of their conclusion that Tamiflu reduced risk of complications from flu?

Only about half of all randomized clinical trials are published, despite regulations requiring publication, and the requirements of the Declaration of Helsinki

Authors have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.

Since the obvious conclusion is that the unpublished studies are less favorable than the published ones, patients and the medical community cannot be sure about the benefits of even the most promising treatments.  The uncertainty always matters at least to a small group of patients, but in the case of Tamiflu it matters to the whole world. The 2009-2010 influenza pandemic was relatively minor, but still killed more than 250000 people worldwide (by most estimates, more than the Iraq war). The 1918 pandemic was at least twenty times worse. Before it happens again, we need to know which treatments work and which do not work.

Via BetterPosters, a site devoted to graphics showing the US rank in international comparisons on a range of things.  Some of the graphs are misleading because they look at totals rather than some per-capita quantity and so the US ranks high because it’s a big country.

Others …well, see for yourselves.

Stuff has a (mostly reasonable) story on alcohol and cancer, quoting Prof Doug Sellman

“The ethanol in alcohol is a group one carcinogen, like asbestos,” 

Many of the readers of this story won’t know what a “group one carcinogen” is.  Given the context, a reader might well assume that “group one carcinogens” are those that carry the largest risks of cancer, or cause the most serious cancers. In fact, all it means is that an additional hazard of cancer, whether high or low, has been definitely established, because that’s all the IARC review process tries to do. The Preamble to the IARC Monographs that define these carcinogens says

A cancer ‘hazard’ is an agent that is capable of causing cancer under some circumstances, while a cancer ‘risk’ is an estimate of the carcinogenic effects expected from exposure to a cancer hazard. The Monographs are an exercise in evaluating cancer hazards, despite the historical presence of the word ‘risks’ in the title. The distinction between hazard and risk is important, and the Monographs identify cancer hazards even when risks are very low at current exposure levels, because new uses or unforeseen exposures could engender risks that are significantly higher.

Group 1 carcinogens tend to either be very common exposures or to cause very specific types of cancer, because those are the two scenarios that make it easy to establish definitely that there is a risk.  They include asbestos, arsenic, sunlight, birth control pills, plutonium, diesel exhaust, and wood dust.

Some group 1 carcinogens, such as tobacco and hepatitis B, are responsible for large numbers of cancer deaths worldwide. Others, such as plutonium and diethylstilbestrol, are responsible only for small numbers of deaths. Some group 1 carcinogens cause aggressive, untreatable tumours; for others, such as human papillomavirus, disease is largely preventable by screening; still others, such as sunlight, sometimes cause serious disease but mostly cause relatively minor tumours.

The phrase “group one carcinogen” is only relevant in an argument over whether the risk is zero or non-zero. Its use in other contexts suggests that someone doesn’t know what it means, or perhaps hopes that you don’t.

Congratulations to Nick Iversen for his nomination of this stat of the week from the NZ Herald:

This story states that “Public opinion has turned …” To justify this they quote statistics from surveys done in 1012 and 2103.

But the surveys are not comparable. They asked different questions.

The 2012 survey questions and results:

I disapprove 40.3%
I approve 19.6%
I conditionally approve 37.7%

The 2013 survey questions and results:

I disapprove 61.5%
I approve 33.8%

When the condition (that the number of pokie machines drops across the city) was removed from the 2013 survey those respondents had to switch to a new answer.

My guess is that most would have switched to “I disapprove” because their approval was conditional and the condition has now been removed. If so then the 2012 survey would have had results:

I disapprove 78.0%
I approve 19.6%

In that case public opinion has moved in the opposite direction to that in the headline.

Even if only half would have switched the figures would have been

I disapprove 59.2%
I approve 38.5%

and it still looks like the opposite to the headline.

The headline is pure fiction. The numbers don’t support it. The two surveys can’t be compared.

For more, see Thomas’ post on this.

More on the ongoing fluoridation story:

Firstly, there is a very good statement from the PM’s chief science adviser, Peter Gluckman.  As he says, the scientific issues are entirely settled: at the concentrations used in treating water, fluoride reduces tooth decay and does not cause any harm.  At one time there was scientific uncertainty about adverse health effects; this is just not the case any more.  There is still a question of whether you want to treat the whole population in this way.  We add iodine to salt and folate to bread, but these prevent more serious illnesses than fluoridation does, and there are fewer people with an irrational fear of iodine or folate.

Second, the Herald has a Digipoll on fluoridation

The poll showed 48 per cent of New Zealanders supported the addition of fluoride – double the 25 per cent of those who opposed its use. A further 24 per cent believed the issue should be left to local councillors to decide.

Unfortunately, the poll tried to use a single question to address two unrelated issues: do you want fluoride in your water?, and should the decision be made nationally or locally?  As a consequence, it’s hard to interpret the results.  The ratio of for:against is about the same as in the Hamilton referendum that started fluoridation there in 2006, but if you assume all the people who want the issue decided locally  are really against fluoridation, the opinion would be nearly 50:50.  It obviously isn’t reasonable to assume everyone in favour of local decision-making is against fluoridation — I’m on record as a counterexample — but there’s no way to know how these folks would split.

The Herald story goes on to quote an antifluoride lobbyist

Ms Byrne said the group had science to back its claims that fluoride was toxic and harmful when added to water and without applying it directly to teeth offered none of the benefits health authorities claimed.

However, that is hotly disputed within the science community.

It’s not disputed within the scientific community, it’s disputed by the scientific community. The science, as Sir Peter observes, is settled.

Each week, we would like to invite readers of Stats Chat to submit nominations for our Stat of the Week competition and be in with the chance to win an iTunes voucher.

Here’s how it works:

• Anyone may add a comment on this post to nominate their Stat of the Week candidate before midday Friday July 5 2013.
• Statistics can be bad, exemplary or fascinating.
• The statistic must be in the NZ media during the period of June 29 – July 5 2013 inclusive.
• Quote the statistic, when and where it was published and tell us why it should be our Stat of the Week.

Next Monday at midday we’ll announce the winner of this week’s Stat of the Week competition, and start a new one.

(more…)

From Stuff

Auckland may have the population, but Taranaki, Southland and the West Coast were the fastest-growing regions in the aftermath of the global financial crisis, new figures show.

That’s true, but if you look at the pretty interactive graphics, the differences in growth are basically restricted  to growth from 2007 to 2008.  Changes since then have been small in all regions, and if anything, worse in Taranaki. For example

In a Stat-of-the-Week nomination, Nick Iversen points out a Herald story about the convention centre deal, under the headline “Support disappears for convention deal”

Public opinion has turned against the Government’s SkyCity international convention centre deal just days before it is due to be signed off, allowing for 230 extra poker machines at the downtown Auckland casino.

The latest Herald-DigiPoll survey shows 61.5 per cent of those polled disapprove of the deal while 33.8 per cent approve.

That’s a sharp turnaround from a year ago when a similar poll found 40.3 per cent disapproved and 57.3 supported it.

As Nick and the SkyCity spokesperson point out, the poll last year had a ‘conditionally approve’ option, so they aren’t really comparable. In fact it’s worse than that. the Herald’s headline last year was “Public opposed to SkyCity deal — or want conditions“, so it’s not just a matter of interpreting last year’s data; the story isn’t even consistent with last year’s headline.

Last year, the poll found 40.3% disapproved, 37.7% approved if the total number of gaming machines across Auckland decreased, and 19.6% approved.   This year 61.5% disapprove and 33.8% approve. It’s hard to decide how the 37.7% of conditional approvals should be apportioned, since the number of machines is going down, but it’s going down for reasons basically unrelated to the SkyCity deal.

Some possibilities

• All the conditional approvals should be treated as approvals. That’s the Herald’s current interpretation and gives 57.3% approval last year, and a roughly 25% decrease in approval
• All the conditional approvals should be treated as disapprovals. That was the Herald’s interpretation last year, and gives 19.6% approval last year and and a roughly 13% increase in approval
• The conditional approvals should be ignored, and we should look at the proportion of approval among those approving or disapproving.  That gives 32.7% approval last year, almost identical to this year.

It’s clear that you really can’t say anything very useful about the change using the two different sets of questions.  On the other hand, it’s also clear that the majority in Auckland is currently opposed to the deal, and if public opinion was relevant to the decision-making process, that, rather than the change since last year, would be the important fact.

Sense about Science (a British charity whose name, unusually, is actually accurate) have just launched a publication “Making Sense of Uncertainty”, following their previous guides for the public and journalists that cover screening, medical tests, chemical stories, statistics, and radiation.

Researchers in climate science, disease modelling, epidemiology, weather forecasting and natural hazard prediction say that we should be relieved when scientists describe the uncertainties in their work. It doesn’t necessarily mean that we cannot make decisions – we might well have ‘operational knowledge’ – but it does mean that there is greater confidence about what is known and unknown.
Launching a guide to Making Sense of Uncertainty at the World Conference of Science Journalists today, researchers working in some of the most significant, cutting edge fields say that if policy makers and the public are discouraged by the existence of uncertainty, we miss out on important discussions about the development of new drugs, taking action to mitigate the impact of natural hazards, how to respond to the changing climate and to pandemic threats.
Interrogated with the question ‘But are you certain?’, they say, they have ended up sounding defensive or as though their results are not meaningful. Instead we need to embrace uncertainty, especially when trying to understand more about complex systems, and ask about operational knowledge: ‘What do we need to know to make a decision? And do we know it?’ 

From the World Conference of Science Journalists, via @roobina (Ruth Francis), ten tweets on reporting clinical trials

1. Was this #trial registered before it began? If not then check for rigged design, or hidden negative results on similar trials.
   
2. Is primary outcome reported in paper the same as primary outcome spec in protocol? If no report maybe deeply flawed.
3. Look for other trials by co or group, or on treatment, on registries to see if it represents cherry picked finding
4. ALWAYS mention who funded the trial. Do any of ethics committee people have some interest with the funding company
5. Will country where work is done benefit? Will drug be available at lower cost? Is disorder or disease a problem there
6. How many patients were on the trial, and how many were in each arm?
7. What was being compared (drug vs placebo? Drug vs standard care? Drug with no control arm?
8. Be precise about people/patient who benefited – advanced disease, a particular form of a disease?
9. Report natural frequencies: “13 people per 10000 experienced x”, rather than “1.3% of people experienced x”
10. NO relative risks. Paint findings clearly: improved survival by 3%: BAD. Ppl lived 2 months longer on average: GOOD

Who says you can’t say anything useful in 140 characters?