December 19, 2013

Difficulties in interpreting rare responses in surveys

If some event is rare, then your survey sample won’t have many people who truly experienced it, so even a small rate of error or false reporting will overwhelm the true events, and can lead to estimates that are off by a lot more than the theoretical margin of sampling error.

The Herald has picked up on one of the other papers (open access, not linked) in this year’s Christmas BMJ, which looks at data from the National Longitudinal Study of Youth, in the US. This is an important social and health survey, and the paper is written completely seriously. Except for the topic

Of 7870 eligible women, 5340 reported a pregnancy, of whom 45 (0.8% of pregnant women) reported a virgin pregnancy (table 1). Perceived importance of religion was associated with virginity but not with virgin pregnancy. The prevalence of abstinence pledges was 15.5%. The virgins who reported pregnancies were more likely to have pledged chastity (30.5%) than the non-virgins who reported pregnancies (15.0%, P=0.01) or the other virgins (21.2%, P=0.007).

and

A third group of women (n=244) not included in analysis, “born again virgins,” reported a history of sexual intercourse early in the study but later provided a conflicting report indicating virginity. Reports of pregnancy among born again virgins were associated with greater knowledge of contraception methods with higher failure rates (withdrawal and rhythm methods) and lower interview quality (data not shown), and reports from this group may be subject to greater misclassification error.

The survey had carefully-designed and tested questions, and used computer-assisted interviewing to make participants more willing to answer potentially embarrassing questions. It’s about as good as you can get. But it’s not perfect.

avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »

Comments

  • avatar
    Ken Rice

    “But it’s not perfect.”

    You mean it’s not immaculate, surely?

    1 year ago Reply

    • avatar
      Thomas Lumley

      The paper goes to some lengths to explain that they aren’t doing that joke because it’s actually a theological confusion: immaculate conception is a strictly Catholic dogma about Mary’s birth.

      1 year ago Reply

  • avatar
    megan pledger

    It’s most likely mis-keying by the teens at data entry … but abstinance pledges tend to be about being a virgin (not engaging in vaginal intercourse) on getting married but doesn’t rule out a lot of other sexual contact, some of which has some (small) risk of pregnancy. I wouldn’t be surprised if there were some genuine cases.

    1 year ago Reply

    • avatar
      Thomas Lumley

      They give reasons why it probably isn’t just data entry — the result remains consistent over multiple survey waves for some women.

      1 year ago Reply

Add a comment

First time commenting? Please use your real first name and surname and read the Comment Policy.