American football games, like many sporting events, start with a coin toss, in this case to decide which team is playing in which direction. At the last 14 Superbowls, the team from the National Football Conference has won the toss (via). In a standard test of the hypothesis that the coin was fair, the p-value would be 0.0001. So, does this mean the NFC is cheating? Well, no. We have overwhelmingly good reasons to believe that coin tosses are very close to fair, and a mere 1 in 8000 coincidence shouldn’t change our minds. As Tom Stoppard put it in Rosencrantz and Guildensten Are Dead: “A spectacular vindication of the principle that each coin, spun individually, is just as likely to come up head as tails, and should cause no surprise each individual time it does.”
The generalization of this principle to studies purporting to find small, but statistically significant, benefits of homeopathy is left as an exercise to the reader.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »