‘Close’ counts in horseshoes and clinical trials
Elections are designed to produce a result: someone wins. The losing side doesn’t get to enact a fair share of their program just for getting close.
Clinical trials aren’t like that. They do feed into regulatory decisions, which may be yes/no in a similar way, but when we talk about what was learned in a trial it isn’t just binary, or shouldn’t be. Unfortunately, there’s a tendency for “the trial did not provide convincing evidence of mortality reductions” to get simplified to “the trial did not provide any evidence of mortality reductions” and then to “the treatment does not reduce mortality”.
The NIH trial of remdesivir for Covid was a typical example. Here’s the result.
The estimated reduction is about 25% — not a cure, but quite worthwhile if true. There’s a lot of uncertainty: the trial data are consistent with there being no benefit (a ratio of 1; the vertical line) and equally consistent with a 50% mortality reduction. The results still got described as “remdesivir does not reduce mortality”.
This matters, because we need to distinguish an inconclusive but moderately favourable result from the other sort of negative trial. This week we got results from the larger SOLIDARITY trial, coordinated by the WHO. Here’s a comparison of the two
The two trials are clearly consistent with each other, but the uncertainty around the results is a lot smaller with the WHO trial. Importantly, the estimated benefit in the WHO trial is a much less impressive 5% reduction, and the lower limit is about 20%. The WHO trial can reasonably be summarised as saying ‘there is no substantial reduction in mortality with remdesivir’, and it wouldn’t be too much of a stretch to say ‘remdesivir doesn’t reduce mortality’ (in patients similar to these ones).
We can combine the two sets of information:
The diamond is the uncertainty interval for the combination of the two trials (it’s a different shape so you don’t think it’s a third trial). It mostly follows the larger WHO trial, where most of the information is, but it’s shifted a little to the left because of the more positive information from the NIH trial. The takeaway point is the same as from just the WHO trial, though: remdesivir doesn’t reduce mortality much, if at all.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
Fair enough summary. But you would have to admit that the overriding tendency is to over-claim on test results, particularly in the context of the COVID pandemic. So, perhaps more restraint, rather than less, is in order!
4 years ago