The methods behind the statistics do matter
From the US 6th Circuit Court of Appeals (PDF), in a lawsuit alleging false advertising by a US law school, based on a low-quality survey of graduates
For example, the Employment Report for 2010 states that the “average starting salary for all graduates” was $54,796. On its face, the phrase “all graduates” means just that: all Cooley graduates—not just the ones who responded to the survey—made, on average, $54,796. One could assume that, because there were 934 graduates, the average starting salary for all 934 graduates was $54,796. The title of the document containing this statement is “Employment Report and Salary Survey.” Therefore, it cannot be that the average starting salary of all 2010 graduates was $54,796, because the document, entitled “Employment Report and Salary Survey” (emphasis added) was not based on the responses of all of the Cooley graduates in 2010; rather, the document states that the number of 2010 graduates was 934, but the number of graduates with employment status known was 780. So, the “[a]verage starting salary for all graduates” would instead mean the average starting salary of graduates who responded to the survey and chose to include their salary information—not the average salary of all Cooley graduates in any given year.
We agree with the district court that this statistic is “objectively untrue,” but that the graduates’ reliance upon it was “also unreasonable,” which dooms their fraudulent misrepresentation claim.
It’s not just statisticians who think you need to pay attention to where the numbers come from.
(via)
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »