Use this new treatment while it still works

An analysis reported in JAMA, and more accessibly by the LA Times shows that really dramatic beneficial effects of new treatments, even in good-quality clinical trials, are usually over-estimates.

They looked at 85,000 treatments for which there were multiple trials and binary (yes/no) outcomes measured.  In the 8000 treatments where the first trial showed a very large benefit, 90% of the time the subsequent trials showed a smaller benefit.

Also, when  large benefits were seen in well-replicated studies these were not typically for life or death issues: the researchers say

These include clinical benefits such as control of nocturnal enuresis with alarms in children, or symptomatic improvement with 5-aminosalicylic acid in ulcerative colitis; mostly mild or modest harms such as burning with capsaicin or local tenderness with the influenza vaccine; laboratory-determined response such as induction of hepatitis B surface antigen with hepatitis B vaccination; and control of acute pain with analgesics such as etoricoxib or diclofenac. As shown, all of these effects corresponded to very large absolute risk differences.

These findings aren’t actually surprising: if a treatment does very well in its first clinical trial it was probably lucky as well as effective.  Very large benefits in survival are very rare, so if you think you’re seeing one, you’re probably wrong .  It’s still useful for someone to put together all the data confirming what we expect, and it’s useful to remember when you see a headline about miracle cures.