June 4, 2013

Survey respondents are lying, not ignorant

At least, that’s the conclusion of a new paper from the National Bureau of Economic Research.

It’s a common observation that some survey responses, if taken seriously, imply many partisans are dumber than a sack of hammers.  My favorite example is the 32% of respondents who said the Gulf of Mexico oil well explosion made them more likely to support off-shore oil drilling.

As Dylan Matthews writes in the Washington Post, though, the research suggests people do know better. Ordinarily they give the approved politically-correct answer for their party

In the control group, the authors find what Bartels, Nyhan and Reifler found: There are big partisan gaps in the accuracy of responses. …. For example, Republicans were likelier than Democrats to correctly state that U.S. casualties in Iraq fell from 2007 to 2008, and Democrats were likelier than Republicans to correctly state that unemployment and inflation rose under Bush’s presidency.

But in an experimental group where correct answers increased your chance of winning a prize, the accuracy improved markedly:

Take unemployment: Without any money involved, Democrats’ estimates of the change in unemployment under Bush were about 0.9 points higher than Republicans’ estimates. But when correct answers were rewarded, that gap shrank to 0.4 points. When correct answers and “don’t knows” were rewarded, it shrank to 0.2 points.

This is probably good news for journalism and for democracy.  It’s not such good news for statisticians.

avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »

Comments

  • avatar
    megan pledger

    I don’t think they lie. I think they want to get the survey over and done with as quickly as possible so they grab an answer off the top of their head witht the underlying bias that Bush=good/Bush=bad depending on party affiliation. When there is a prize at stake they are willing to invest more time in getting the right answer by going through their memories more rigorously.

    2 years ago Reply

  • avatar
    Thomas Lumley

    That’s possible. I don’t think the experiment reported time taken.

    2 years ago Reply

  • avatar

    It just goes to prove something the PR industry has been making good use of for over a decade – Bribery Works.

    2 years ago Reply

    • avatar
      Thomas Lumley

      Remarkably small amounts of bribery, though. The expected return was about US$1/correct answer, quite a bit less than the per-person cost of telephone sampling.

      2 years ago Reply

Add a comment

First time commenting? Please use your real first name and surname and read the Comment Policy.