Stats Chat

From the Herald

A new scientific test is able to detect which 14-year-olds will become binge drinkers by the time they hit 16.

A study published in the journal Nature describes how scientists have developed a system that weighs up a range of risk factors and predicts – with about 70 per cent accuracy – which teens will become heavy drinkers.

That’s true, but the definition of accuracy is doing quite a bit of work here.

We don’t have figures for 16 year olds, but according to the Ministry of Health about 20% of 15-17 year olds have ‘hazardous drinking patterns.’ That means I can predict with 80% accuracy without even needing to weigh up a range of risk factors — I just need to predict “No” each time. Parents, teachers, or people working with youth could probably do better than my 80% accuracy.

The researchers found that their test correctly classified 73% of the non-binge-drinker and 67% of the binge drinkers, which means it would get 72% of people classified correctly. That’s rather worse than my trivial “the kids are ok” predictor. In order to be any use, the new test, which combines brain imaging and detailed interviews, needs to be set to a higher threshold, so it predicts fewer drinkers.  The researchers could have done this, but they didn’t.

Also, in order to be any use, the test needs to identify a group who will selectively benefit from some feasible intervention, and there needs to be funding to supply both this intervention, and the cost of doing long interviews and fMRI brain imaging on large groups of teenagers. And that needs to be the best way to spend the money.

Team Ratings for Round 17

The basic method is described on my Department home page. I have made some changes to the methodology this year, including shrinking the ratings between seasons.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 117 matches played, 67 of which were correctly predicted, a success rate of 57.3%.

Here are the predictions for last week’s games.

Predictions for Round 17

Here are the predictions for Round 17. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 18

The basic method is described on my Department home page. I have made some changes to the methodology this year, including shrinking the ratings between seasons.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 106 matches played, 67 of which were correctly predicted, a success rate of 63.2%.

Here are the predictions for last week’s games.

Predictions for Round 18

Here are the predictions for Round 18. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

The official maximum margin of error for an election poll with a simple random sample of 1000 people is 3.099%. Real life is more complicated.

In reality, not everyone is willing to talk to the nice researchers, so they either have to keep going until they get a representative-looking number of people in each group they are interested in, or take what they can get and reweight the data — if young people are under-represented, give each one more weight. Also, they can only get a simple random sample of telephones, so there are more complications in handling varying household sizes. And even once they have 1000 people, some of them will say “Dunno” or “The Conservatives? That’s the one with that nice Mr Key, isn’t it?”

After all this has shaken out it’s amazing the polls do as well as they do, and it would be unrealistic to hope that the pure mathematical elegance of the maximum margin of error held up exactly.  Survey statisticians use the term “design effect” to describe how inefficient a sampling method is compared to ideal simple random sampling. If you have a design effect of 2, your sample of 1000 people is as good as an ideal simple random sample of 500 people.

We’d like to know the design effect for individual election polls, but it’s hard. There isn’t any mathematical formula for design effects under quota sampling, and while there is a mathematical estimate for design effects after reweighting it isn’t actually all that accurate.  What we can do, thanks to Peter Green’s averaging code, is estimate the average design effect across multiple polls, by seeing how much the poll results really vary around the smooth trend. [Update: this is Wikipedia’s graph, but I used Peter’s code]

I did this for National because it’s easiest, and because their margin of error should be close to the maximum margin of error (since their vote is fairly close to 50%). The standard deviation of the residuals from the smooth trend curve is 2.1%, compared to 1.6% for a simple random sample of 1000 people. That would be a design effect of (2.1/1.6)², or 1.8.  Based on the Fairfax/Ipsos numbers, about half of that could be due to dropping the undecided voters.

In principle, I could have overestimated the design effect this way because sharp changes in party preference would look like unusually large random errors. That’s not a big issue here: if you re-estimate using a standard deviation estimator that’s resistant to big errors (the median absolute deviation) you get a slightly larger design effect estimate.  There may be sharp changes, but there aren’t all that many of them, so they don’t have a big impact.

If the perfect mathematical maximum-margin-of-error is about 3.1%, the added real-world variability turns that into about 4.2%, which isn’t that bad. This doesn’t take bias into account — if something strange is happening with undecided voters, the impact could be a lot bigger than sampling error.

From the Herald (via @BKDrinkwater on Twitter)

Wages have only gone up $34.53 annually against house prices, which are up by $38,000.

These are the findings of the Home Affordability Report quarterly survey released by Massey University this morning.

At face value, that first sentence doesn’t make any sense, and also looks untrue. Wages have gone up quite a lot more than $34.53 annually. It is, however, almost a quote from the report, which the Herald embeds in their online story

From Deadspin. No further comment needed.

The discussion over the Facebook experiment seems to involve a lot of people being honestly surprised that other people feel differently.

One interesting correlation based on my Twitter feed is that scientists involved in human subjects research were disturbed by the research and those not involved in human subjects research were not. This suggests our indoctrination in research ethics has some impact, but doesn’t answer the question of who is right.

Some links that cover most of the issues

• Michelle Meyer: How an IRB Could Have Legitimately Approved the Facebook Experiment—and Why that May Be a Good Thing
• Coding Coduct: Frame Clashes, or: Why the Facebook Emotion Experiment Stirs Such Emotion
• John Foreman: Facebook’s solution to big data’s “content problem:” dumber users
• Tal Yarkoni: In defense of Facebook
• Zeynep Tufekci: Facebook and Engineering the Public
• Adam Kramer (one of the people who did it)
• (update) Tyler Cowen: Should we care that Facebook is manipulating us

Each week, we would like to invite readers of Stats Chat to submit nominations for our Stat of the Week competition and be in with the chance to win an iTunes voucher.

Here’s how it works:

• Anyone may add a comment on this post to nominate their Stat of the Week candidate before midday Friday July 4 2014.
• Statistics can be bad, exemplary or fascinating.
• The statistic must be in the NZ media during the period of June 28 – July 4 2014 inclusive.
• Quote the statistic, when and where it was published and tell us why it should be our Stat of the Week.

Next Monday at midday we’ll announce the winner of this week’s Stat of the Week competition, and start a new one.

(more…)

• Motivated reasoning: In a survey of doctors “68% of the 125 circumcised males supported circumcision, and 69% of the 106 uncircumcised males opposed circumcision“
• “And although many would consider this creepy stalking, that’s not actually my problem with it.” Cathy O’Neill on medical data mining
• From Pacific Standard magazine: How to get people to donate. “One [study] finds that requests featuring a single child are more effective than those with multiple faces. A second suggests that child had better not be too attractive.”
• A lighter side: colour dictionaries — what scientists did before Pantone

When you read “The university did not reveal how the study was carried out” in a news story about a research article, you’d expect the story to be covering some sort of scandal. Not this time.

The Herald story  is about broccoli and asthma

They say eating up to two cups of lightly steamed broccoli a day can help clear the airways, prevent deterioration in the condition and even reduce or reverse lung damage.

Other vegetables with the same effect include kale, cabbage, brussels sprouts, cauliflower and bok choy.

…

Using broccoli to treat asthma may also help for people who don’t respond to traditional treatment.

‘How the study was carried out’ isn’t just a matter of detail: if they just gave people broccoli, they wouldn’t know what other vegetables had the same effect, so maybe it wasn’t broccoli but some sort of extract? Was it even experimental or just observational? And did they actually test people who don’t respond to traditional treatment? And what exactly does that mean — failing to respond is pretty rare, though failing to get good control of asthma attacks isn’t.

The Daily Mail story was actually more informative (and that’s not a sentence I like to find myself writing). They reported a claim that wasn’t in the press release

The finding due to sulforaphane naturally occurring in broccoli and other cruciferous vegetables, which may help protect against respiratory inflammation that can cause asthma.

Even then, it isn’t clear whether the research really found that sulforaphane was responsible, or whether that’s just their theory about why broccoli is effective. 

My guess is that the point of the press release is the last sentence

Ms Mazarakis will be presenting the research findings at the 2014 Undergraduate Research Conference about Food Safety in Shanghai, China.

That’s a reasonable basis for a press release, and potentially for a story if you’re in Melbourne. The rest isn’t. It’s not science until they tell you what they did.