Measuring what you care about
From the Twitters
Some people are suggesting the 99.9% bus punctuality stats might be made up. I am shocked. Truly shocked.
— AkTransportBlog (@AkTransportBlog) August 21, 2013
There followed a lively discussion, with comparisons to election results in various unpleasant places.
As I’ve suggested before, the really damning aspect of the 99.9% bus punctuality stats is that they might actually be honest. That is, `punctuality’ has been defined in a way that it does not reflect passenger experience and is of almost no interest to passengers. A bus is ‘punctual’ if it starts its route (which I assume means the driver starts up the ticket system) no more than 5 minutes late. What happens later doesn’t matter, as long as the bus does eventually reach its destination.
The real-time prediction system (although it may not be great at predicting the future) knows where the buses have been, so it would be feasible to set up a punctuality summary that actually measured punctuality — perhaps the proportion of major timepoints where the bus was no more than 5 minutes late or 1 minute early in departing.
I mention this point again because I’m teaching Design of Clinical Trials at the moment, and one of the big issues is ‘surrogate outcomes’. That is, people running studies are often tempted to measure something convenient rather than what patients care about: diabetes trials may measure blood sugar rather than heart attacks and kidney failure; heart disease trials may measure blood pressure rather than heart attacks or strokes; cancer trials may measure tumour size rather than survival or symptoms. This matters, because there are important examples where a treatment improves a surrogate outcome, but makes the real outcomes worse.
There is no way that the Auckland bus punctuality statistics are accurate measurements of something that matters to passengers. It’s not a matter of life and death as it is in clinical trials, but it’s still a waste of time and money. And since the Auckland bus system is actually pretty good, it makes Auckland Transport and the Council look unnecessarily stupid.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »