March 13, 2013

Is epidemiology 90% wrong?

There’s been a recent recurrence of the factoid that 90% of results in epidemiology are wrong. For example, @StatFact on Twitter posted ‘Empirical evidence is that 80-90% of the claims made by epidemiologists are false.’ with a link to a talk by Stanley Young at the National Institute of Statistical Sciences.  I replied “For suitable values of ‘claim’ and ‘false'”, and if you don’t want to read further, that’s a good summary.

The claim is obviously wrong if interpreted at face value: firstly, epidemiologists mostly make claims of the form “in this data set, processed meat consumption was higher in people who ended up dying during follow-up”. These claims are usually true, modulo occasional fraud or error. Certainly 90% of them aren’t false. Secondly, if some papers say people who ate more processed meat were more likely to die and other papers say they weren’t, one or other claim must be true and the whole 90% thing makes no sense.

Interpreted in its strongest sense, then, the claim that 90% of epidemiological results are false is uninterestingly false.  Using this sort of interpretation would obviously be unreasonable.  Unfortunately, it’s this sort of interpretation of results in epidemiology that you need in order to make the 90% statistic true and interesting.

Let’s look at the results and conclusion from the abstract of the processed-meat paper I wrote about last week

Results: As of June 2009, 26,344 deaths were observed. After multivariate adjustment, a high consumption of red meat was related to higher all-cause mortality (hazard ratio (HR) = 1.14, 95% confidence interval (CI) 1.01 to 1.28, 160+ versus 10 to 19.9 g/day), and the association was stronger for processed meat (HR = 1.44, 95% CI 1.24 to 1.66, 160+ versus 10 to 19.9 g/day). After correction for measurement error, higher all-cause mortality remained significant only for processed meat (HR = 1.18, 95% CI 1.11 to 1.25, per 50 g/d). We estimated that 3.3% (95% CI 1.5% to 5.0%) of deaths could be prevented if all participants had a processed meat consumption of less than 20 g/day. Significant associations with processed meat intake were observed for cardiovascular diseases, cancer, and ‘other causes of death’. The consumption of poultry was not related to all-cause mortality. 

Conclusions: The results of our analysis support a moderate positive association between processed meat consumption and mortality, in particular due to cardiovascular diseases, but also to cancer.

So, the claims in the results section are about observed differences in a particular data set, and presumably are true.  The claim in the conclusion is that this ‘supports’ ‘an association’.  If you interpret the conclusion as claiming there is definitive evidence of an effect of processed meat, you’re looking at the sort of claim that is claimed to be 90% wrong. Epidemiologists don’t interpret their literature this way, and since they are the audience they write for, their interpretation of what they mean should at least be considered seriously.

So, how good is the evidence that 90% of epidemiology results interpreted this way are false?  It depends.  The argument is that most hypotheses about effects are wrong, and that the standard for associations used in epidemiology is not a terribly strong filter, so that most hypotheses that survive the filter are still wrong.  That’s reasonably as far as it goes.  It does depend on taking studies in isolation.  In this example there are both previous epidemiological studies and biochemical evidence to suggest that fat, salt, smoke, and nitrates from meat curing might all be harmful. In other papers the background evidence can vary from strongly in favor to strongly against, and this needs to be taken into account.

A more moderate and reasonable claim about epidemiological studies is that very few of them provide information that should make you change your behaviour.  This applies to the good ones as well as the bad ones.

avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »

Comments

  • avatar

    Nice post! I love your conclusion! Perhaps another way to think about all this is that a large fraction (no idea of the actual fraction) of epidemiological studies are not replicated by others. This of course is true of genetic studies as well. Does this mean they are wrong? Not necessarily, but how do you know, when 1) every observation is unique, 2) the complex diseases of interest these days are, well, complex, and 3) it’s impossible to precisely replicate a study of factors with small effects, as so many risk factors including genes are with respect to complex diseases, even in the same population? And anyway, they’re small effects, so hard to distinguish signal from noise. Yep, very few provide information that should make you change your behavior!

    11 years ago

  • avatar
    Joseph Delaney

    “more moderate and reasonable claim about epidemiological studies is that very few of them provide information that should make you change your behaviour. This applies to the good ones as well as the bad ones.”

    Ironically, this can be true even if you actually believe the claims of the paper are true and the suggested intervention is actionable. For example, there are many studies that suggest that cigarette smoking does not promote good health. We are reasonably confident there is a link with an increased risk of lung cancer and/or cardiovascular disease. Another study that showed that cigarette smoking was associated with an increase in varicose veins isn’t really likely to change the decision matrix here if you are still engaged in the activity.

    So the real tipping points have got to be rare.

    11 years ago

  • avatar

    Mrs Cooper: The trick to pancakes is bacon grease. I cook everything in it.

    Leonard: Everything? Aren’t you worried about your health?

    Mrs Cooper: Oh, doctors are always changing their mind. One week bacon grease is bad for you. The next week we’re not getting enough of it.

    11 years ago