When are self-selected samples worth discussing?
From recent weeks, three examples of claims from self-selected samples:
- The Dominion Post: “Women still want spouse to earn more”
- The New Zealand Association of Scientists survey on perceptions of the National Science Challenges
- Dr Kate Clancy’s survey of sexual harassment in academic fieldwork, from PLoS One
In all three cases, you’d expect the pattern to generalise to some extent, but not quantitatively. The dating site in question specifically boasts about the non-representativeness of its members; the NZAS survey was sent to people who’d be likely to care, and there wasn’t much time to respond; scientists who had experienced or witnessed harassment would be more likely to respond and to pass the survey along to others.
I think two of these are worth presenting and discussing, and the other one isn’t, and that’s not just because two of them agree with my political prejudices.
The key question to ask when looking at this sort of probably non-representative sample, is whether the response you see would still be interesting if no-one outside the sample shared it. That is, the surveys tell us at a minimum
- there exist 350 women in New Zealand who wouldn’t marry a man earning less than them, and are prepared to say so
- there exist 200-odd scientists in NZ who think the National Science Challenges were badly chosen or conducted, and are prepared to say so
- there exist 417 scientists who have experienced verbal sexual harassment, and 139 who have experienced unwanted physical contact from other research staff during fieldwork, and are prepared to say so.
I would argue that the first of these is completely uninteresting, but the second is contrary to the impressions being given by the government, and the third should worry scientists who participate in or organise fieldwork.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
This is a great take on the issue. My objection to the National Science Challenge survey coverage was the focus on the dubious “80% of scientists say…” claim. While 80% is a big number, and 200 could be seen as a relatively small number, as you say, it is that absolute number which is actually more important. That number of scientists united on an issue has immensely more credibility than a highly biased proportion.
Cheers
10 years ago
This is indeed a good way to frame the survey, and good on Nicola Gaston et al. for organizing the survey in such short notice.
However, I wonder if the NZAS would have gotten more mileage out of directly surveying their membership. This could of course still be done IMHO.
Although all things are easy to the one who doesn’t have to do it: this would have been almost as easy as the open surveymonkey url, and prevented the “unscientific survey” criticism. With a small followup of non-responders, it could have answered what proportion of this organization-representative of NZ researchers-felt what. Pity at the moment it seems to have been completely brushed off.
10 years ago
They are looking for funds to do a proper survey of NZ scientists in general, not just their membership (and not just on this issue, I assume)
10 years ago