Vaccine transparency

Moderna (big long PDF) and Pfizer (big long  PDF) have released their clinical trial protocols.  These specify how  to run the trial: the statistical design, and also a lot of other things  like how  they collect adverse events. There’s a story at Reuters and at Buzzfeed. I’m not the right person to  comment about the detailed collection of safety data, except to say that they do describe how they do it,  so you can ask your favourite immunologist or virologist for an opinion.

The statistical design of early stopping is reasonable in both trials, as you’d hope. Both trials will end up declaring a positive result with an estimated vaccine efficacy  greater than 0.5, and with  0.3 efficacy outside the margin of error.   If the true efficacy is 0.6 they have a  90% chance of  declaring a positive result.

They vary in how they place their bets for the case where the vaccine is more effective than that — and it might be, who  knows?.  If the vaccine is actually 90% effective, you don’t need to test it on as many people, and the urgency of knowing it works is greater. The trials are designed so  they can stop early, but still preserve their overall standards for statistical evidence.

The tradeoff for stopping  earlier is that you have less detailed  information: you  know the vaccine is  good, but you have less precise estimates of how good, and you have somewhat less safety information because you’ve vaccinated fewer people or followed them for less time.  There is an important and nerdy statistical literature on the  pros and cons of basically any possible set of guidelines for stopping early. If you’re interested, this is a good starting place.

Moderna designed their trial more or less the way I would have done, with the so-called O’Brien-Fleming guidelines, which are standard and pretty conservative about early stopping. Pfizer are less conservative: if the vaccine is very effective, they have more chance of  stopping early.   But while they are less conservative, the design is still well in the  range of standard custom and practice.

We still don’t have any idea what the results will be at the early analyses, either for effectiveness or safety.  We don’t know what the US  government will decide to do with the results. But we do know that if the trials follow their protocols, and if the trials stop early, and if there aren’t  prohibitive safety concerns, then we will have good  evidence that  the vaccine works.   We  also will be able to tell if  the companies or  the FDA or anyone changes the standards of evidence after seeing the results.