Posts filed under General (3156)

This just in from National Business Review – thanks to journo Caleb Allison and  NBR Online for giving us permission to upload the content, which sits behind a pay wall.

Commuter survey ‘can’t be trusted’ – statistician 

A statistician questions the validity of a survey promoting flexible working conditions for employees.

A recent survey by Regus – which describes itself as “the world’s largest provider of flexible workspaces” – said 67% of New Zealand employees would spend more time with family if they had a shorter commute as a result of flexible working conditions.

While the survey claimed to have polled more than 16,000 people in 80 countries, NBR ONLINE can reveal the company polled just 54 people in New Zealand.

Auckland University’s Dr Andrew Balemi says while it is a very low number, that alone does not suggest the poll is dodgy.

“Most people obsess about the sample size, but what I obsess about is the sample quality,” Dr Balemi says.

The only way to know if the information is credible is to know how the company undertook the survey.

However, the methodology was not included with the poll.

Dr Balemi says not only does this survey have a small sample size, it doesn’t tell the reader how it was obtained.

“In the absence of any explanation of how they’ve collected the data I wouldn’t trust this information.

“If they can’t even do that, I wouldn’t dignify it with any more consideration.”

He says the company may have a valid methodology and the poll could be worthy, but they should have included that information in the survey.

This follows another recent example of dodgy polling by the Auckland Council.

A press release claiming 63% of Aucklanders favour mayor Len Brown’s city rail loop turned out to have surveyed only 112 people.

These are from the UK Science Media Centre, part of its submission to the Leveson Inquiry

• State the source of the story – e.g. interview, conference, journal article, a survey from a charity or trade body, etc. – ideally with enough information for readers to look it up or a web link.
•  Specify the size and nature of the study – e.g. who/what were the subjects, how long did it last, what was tested or was it an observation? If space, mention the major limitations.
• When reporting a link between two things, indicate whether or not there is evidence that one causes the other.
• Give a sense of the stage of the research – e.g. cells in a laboratory or trials in humans – and a realistic time-frame for any new treatment or technology.
•  On health risks, include the absolute risk whenever it is available in the press release or the research paper – i.e. if ’cupcakes double cancer risk’ state the outright risk of that cancer, with and without cupcakes.
•  Especially on a story with public health implications, try to frame a new finding in the context of other evidence – e.g. does it reinforce or conflict with previous studies? If it attracts serious scientific concerns, they should not be ignored.
•  If space, quote both the researchers themselves and external sources with appropriate expertise. Be wary of scientists and press releases over-claiming for studies.
• Distinguish between findings and interpretation or extrapolation; don’t suggest health advice if none has been offered.
•  Remember patients: don’t call something a ’cure’ that is not a cure.
• Headlines should not mislead the reader about a story’s contents and quotation marks should not be used to dress up overstatement

This blog would be a lot less interesting (except to rugby fans) if the media followed these guidelines.  I think the second-last item is the only one that hasn’t been the basis for one or more posts.     (via)

The American Statistical Association gives an annual award for Excellence in Statistical Reporting.  This year it goes to Amanda Cox, a graphics editor at the New York Times, some of whose graphs we’ve highlighted on the blog.  Here are some more examples and talks.

The award was created to encourage and recognize members of the communications media who have best displayed an informed interest in the science of statistics and its role in public life. The award can be given for a single statistical article or for a body of work. In selecting the recipient, consideration is given to:

• Correctness, clarity, fairness, brevity, and professionalism of the communication
• Importance, relevance and overall effectiveness in impacting the intended audience
• Impact on the growth and national or regional exposure of statistics
• Appreciation and emphasis of the statistical aspects of a particular issue or event
• Excellent coverage of research on statistics or statistical issues

Here’s a different statistical take on the Euro-zone crisis:

Debt crisis: Italy’s statisticians threaten ‘stats black-out’

Italy’s official statisticians are threatening to down calculators and stop reporting on its stricken economy – as they themselves fall victim to the recession they are paid to track. 

Read the details here.

This appeared on stuff.co.nz and in Fairfax papers last week:

After a harrowing trial that gripped the nation, a survey has revealed just one in five New Zealanders think Ewen Macdonald did not murder his brother-in-law Scott Guy.

A jury of 11 handed down a not guilty verdict to Macdonald, 32, last week, after a month-long trial in the High Court at Wellington.

But results to be made public by market research company UMR today show just 20 per cent of people surveyed agreed with Ewen Macdonald being acquitted of slaying Mr Guy outside his rural Feilding home in July 2010.

Living in New Zealand means agreeing to deal with criminal allegations transparently in the courtroom, not the court of (ill-informed, speculative) public opinion. The only people with the information on which to make an informed opinion are members of the jury – and they have delivered a verdict that police will not appeal.  What was UMR thinking?

A question on my recent post about political opinion polls asks

– at what point does the trend become relevant?

– and how do you calculate the margin of error between two polls?

Those are good questions, and the reply was getting long enough that I decided to promote it to a post of its own. The issue is that proportions will fluctuate up and down slightly from poll to poll even if nothing is changing, and we want to distinguish this from real changes in voter attitudes — otherwise there will be a different finding every month and it will look as if public opinion is bouncing around all over the place.  I don’t think you want to base a headline on a difference that’s much below the margin of error, though reporting the differences is fine if you don’t think people can find the press release on their own.

The (maximum) margin of error, which reputable polls usually quote, gives an estimate of uncertainty that’s designed to be fairly conservative. If the poll is well-designed and well-conducted, the difference between the poll estimate and the truth will be less than the maximum margin of error 95% of the time for true proportions near one-half, and more often than 95% for smaller proportions.  The difference will be less than half the margin of error about two-thirds of the time, so being less conservative doesn’t let you shrink the margin very much.   In this case the difference was well under half the margin of error.  In fact, if there were no changes in public opinion you would still see month-to-month differences this big about half the time.

For trends based on just two polls, the margin of error is larger than for a single poll, because it could happen by chance that one poll was a bit too low and the other was a bit too high: the difference between the two polls can easily be larger than the difference between either poll and the truth.

The best way to overcome the random fluctuations to pick up small trends is to do some sort of averaging of polls, either over time, or over competing polling organisations.  In the US, the website fivethirtyeight.com combines all the published polls to get estimates and probabilities of winning the election, and they do very well in short-term predictions.  Here’s a plot for Australian (2007) elections, by Simon Jackman, of  Stanford, where you can see individual poll results (with large fluctuations) around the average curve (which has much smaller uncertainties).  KiwiPollGuy  has apparently done something similar for NZ elections (though I’d be happier if their identity or their methodology was public).

So, how are these numbers computed?  If the poll was a uniform random sample of N people, and the true proportion was P, the margin of error would be 2 * square root(P*(1-P)/N).  The problem then is that we don’t know P — that’s why we’re doing the poll. The maximum margin of error takes P=0.5, which gives the largest margin of error, and one that’s pretty reasonable for a range of P from, say, 15% to 85%. The formula then simplifies to 1/square root of N.   If N is 1000, that’s 3.16%, for N=948 as in the previous post, it is 3.24%.

Why is it  2 * square root(P*(1-P)/N)?  Well, that takes more maths than I’m willing to type in this format so I’m just going to mutter “Bernoulli” at you and refer you to Wikipedia.

For trends based on two polls, as opposed to single polls, it turns out that the squared uncertainties add, so the square of the margin of error for the difference is twice the square of the margin of error for a single poll.  Converting back to actual percentages, that means the margin of error for a difference based on two polls is 1.4 times large than for a single poll.

In reality, the margins of error computed this way are an underestimate, because of non-response and other imperfections in the sampling, but they don’t do too badly.

If you don’t like StatsChat you will probably not enjoy “More or Less”,  a BBC radio show/podcast/blog on statistics in the British media, presented by Tim Harford.  Their new season starts this week.

NBR’s editor Nevil Gibson links up to Stats Chat today in his discussion of the minimum price of alcohol:

In StatsChat’s brilliant daily skewing of journalistic bloopers, University of Auckland Biostatistics Professor Thomas Lumley says this is the opposite point of minimum unit pricing, as opposed to increased excise rates.

Read the full editorial »

How do genetic variants affect biology and health? Is personalised medicine just around the corner?

Thomas Lumley, prolific statschat.org.nz contributor and University of Auckland biostatisician, gives a primer to Kim Hill and her listeners on Radio New Zealand.

As we know, women are paid less than men, and it’s not primarily because of sick-leave differences.   So what does cause it?  Motivated by current US legislation, the Washington Post digs out a 2007 research report that estimates how much of the difference can be statistically explained by factors such as choice of occupation and leaving the workforce to take care of children.

In a sense, this attempts to give a lower bound on the impact of discrimination —  some of the impact of childcare responsibilities is a hangover from traditional roles, but it’s quite possible that in a perfect world there would still be some gender differences in child care.  Similarly, some of the lower pay in majority-female occupations is probably because these occupations had more women, but it’s hard to say how much of it.

The reseachers, Francine Blau and Lawrence Kahn, looked at US data from 1979, 1989, and 1998 (the report was being prepared and revised for a long time). In 1979 the pay ratio was 63%, but comparing men and women with the same education and experience it was 71%, and additionally controlling for occupation, industry, and union coverage it was 82%.  In 1998 the numbers were 80%, 81%, and 91%.

So, by 1998 about 10% of the US pay difference between men and women was explainable by differences in education and experience, about half of it was explainable by working in different industries or occupations, and the rest was not explainable by anything they measured.    I don’t know if anyone has done a similar analysis for New Zealand, where the differential is quite a bit smaller than in the US.

This is an example of the sort of thing you can only do with good-quality survey data; in this case from the University of Michigan’s Panel Study of Income Dynamics.