Posts written by Thomas Lumley (2645)

You may remember StatsChat’s criticism of data quality and analysis in paper about chocolate and Nobel Prizes from a leading medical journal.  Another leading medical journal, BMJ, traditionally has a Christmas issue with not entirely serious papers, typically based on good-quality silly research. One of the past highlights was the systematic review of randomised trials of parachute use.

This year, there’s a survival analysis of chocolate in hospital wards. Survival analysis is the branch of statistics working with the time until an event happens.  Often the event is death, hence the name ‘survival’, but it could be something else bad, such as a heart attack, or something good, such as finding a job.  If you’re a chocolate, it’s being eaten.

The data are a good fit to a constant hazard of consumption, with a rate of just under 1%/minute.  There isn’t any sign of strong heterogeneity — if some chocolates are preferred to others, the preference is either not strong enough or variable enough between people that no chocolates are safe.

Other papers in the Christmas issue include a semi-serious comparison of stem cell size and structure for mice and whales, and the finding that, in Dublin, people called Brady are more likely to have pacemaker treatment for bradycardia (presumably a multiple comparison issue)

• Less research needed:

“In any case, the current uncertainty about any benefit from helmet wearing or promotion is unlikely to be substantially reduced by further research” – Ben Goldacre and David Spiegelhalter

• The Cahill-Keyes map projection, at Wired. All flat maps of the round world’s imagined corners have to distort, but the Cahill-Keyes map largely confines the distortion to the oceans. (and a Javascript version)

• NZ cartographer Chris McDowall (@fogonwater) “When presented with an unlabelled map depicting a random part of the country, I can identify most places purely on the basis of their shape. But, when I close my eyes, their forms fade away …. The maps on this page are an attempt to translate my head landscapes into cartographic artefacts. I am trying to recreate of what I see when I close my eyes“

I’ve been looking at our links in and out over the past year.  Leaving out Twitter, Facebook, various RSS readers, and Reddit, we get the most traffic from Kiwiblog and Simply Statistics, with consistent appearances from Observational Epidemiology, Lindsay Mitchell, Learn and Teach Statistics, Andrew Gelman, Public Address, Anti-Dismal, and my other blog.

Unsurprisingly, our largest outward traffic is to Stuff and NZ Herald, in about a 2:3 ratio.  Wikipedia is next, including informative technical references such as the Optional Stopping Theorem,  Berkson’s Paradox, and the Asch conformity experiments, and cultural phenomena you might not have encountered, such as the Bechdel test, Satoshi Kanazawa, and ‘Hitler Has Only Got One Ball’.

The other main media sites you clicked through to were the BBC, the Guardian, the Washington Post, and the New York Times.

From today’s Herald (or the very similar story at 3News)

A Wellness in the Workplace survey show sickies taken by people who aren’t really ill are estimated to account for 303,000 lost days of work each year, at a cost of $283 million.

Skipping over the estimate of over $900/day for the average cost of a sickie, this is definitely an example where a link to the survey report and some description of methodology might be helpful. The report says

The survey was conducted during the month of
June 2013. In total, 12 associations took part,
sending it out to a proportion of their members.
In addition, BusinessNZ sent the questionnaire
to a number of its Major Companies Group
members. Respondents were asked to report their
absence data for the 12-month period 1 January to
31 December 2012 and provide details of their policies
and practices for managing employee attendance.

In total, 119 responses were received from entities
across the private and public sectors.

which gives more idea about potential (un)representativeness. But most importantly,  while the survey has real data on numbers of absences and on policies, the information on how likely employees were to take sick leave when not sick was just the opinion of their employers. Unless you work for Santa or the NSA, this is going to have a large component of guesswork.

If you’re an employer, and you want to know whether inappropriate use of sick leave is a problem for your organisation, do you want to rely on your own guesses, or on an average of guesses by an anonymous assortment of 119 other organisations around the country?

[UPDATE: I got the calculations wrong, and was too kind to the MoT and the paper. The time to get good evidence is more like 20 years.]

The Sunday Star-Times has a story saying that the reduction in speed-limit tolerance, which is now on all the time in the summer, hasn’t yet shown evidence of a reduction in road deaths. And they’re right, it hasn’t, despite some special pleading from the Police Minister and Assistant Commissioner David Cliff. We’ve looked at this issue before.

However, it’s also important to ask whether we’d expect to see evidence yet if the policy really worked as promised. The Star-Times goes on to say:

While MOT data shows just 13 per cent of fatal crashes were attributable to speed Land Transport Safety Authority spokesman Andy Knackstedt said there was “a wealth of evidence” that showed even very small reductions in speed led to reductions in fatalities and serious injuries, and that lowering the enforcement tolerance meant lower mean speeds.

So, we should ask whether we’d expect a clear and convincing drop in road deaths if the theory behind the policy was sound. And we wouldn’t.

Let’s see what we should expect if the policy prevented 10% or 20% of the deaths from speeding, which works out to 1.3% or 2.6% of all road deaths.  The number of deaths last year was 286, over 366 days. Under the simplest model for road crash data, a Poisson process that basically assumes different roads and time periods are independent, we can work out how long you’d have to wait to have a 50% chance or an 80% chance of the reduction getting below the margin of error

For a 20% reduction in deaths from speeding it takes about 190 days to have an even-money chance of seeing evidence, and about 390 days to have an 80% chance. For a 10% reduction in deaths from speeding it takes about 380 days for an even-money chance of convincing evidence and 760 days for an 80% chance. Under more complex and realistic models for road deaths it would take longer.

There’s no way that we could expect to see convincing evidence yet, and given the much larger unexplained fall in road deaths in recent years, it will probably never be feasible to evaluate the policy just based on road death counts.  We’re going to have to rely on other evidence: overseas data, information on reductions in speeding, data from non-fatal or even non-injury crashes, and other messier sources.

The referendum preliminary results are in, and as you’d expect, there is a substantial majority for “No”, but it’s also substantially smaller than the number who voted National at the last election. Everyone will be able to say the results support their own views, so I won’t bother.

What is interesting is the relationship between turnout and vote. Before going further, think about what you’d expect. (more…)

Q: Have you read the latest issue of PLoS One?

A: What do you mean? PLoS One doesn’t have “issues”, it’s online-only and publishes papers as soon as they are ready.

Q: Well, have you read the study that led to the headline ‘exam results are influenced by genes, not schools‘, which the Herald says is in the latest issue of PLoS One?

A: Perhaps they mean this paper about heritability of exam results.

Q: Yes, that one. Couldn’t they have just linked to it?

A: My sources tell me linking is harder than it looks.

Q:  Whatever. How did they find out that schools don’t matter?

A: That’s not quite what they found out.

Q: Well, they found 60% of education was due to genes, not schools, didn’t they?

A: What would that even mean?

Q: I thought I got to ask the questions.

A: <sigh>

Q: Ok, what did they find?

A: They found that identical twins had a correlation of about 0.8-0.9 between their exam results, and non-identical twins had a weaker correlation, about 0.5-0.6.  If you assume that the only difference between identical and non-identical twin pairs is that the identical twins share more genes, and that the genetic and non-genetic contributions just add, you can estimate how much of the variation between twins was due to genes and how much is due to environmental factors. And they end up with estimates from 40% to 60% for the genetic part.

Q: How is that different from the headline?

A: The effects aren’t going to be additive in reality — genetics isn’t going to let you pass a British history test if you haven’t ever studied any British history — so the heritability is just a summary of variation in the current population under the current conditions. What the study really finds is that British schools currently differ from each other less than British kids do. If you made a lot of kids from Beijing or Buenos Aires take the British GCSE tests (or vice versa) they’d probably do really badly, and that would definitely be due to their schooling, not their genes.

Q: And what do the researchers say?

A: Pretty much what I said. The Herald quotes them further down in the story

“Since we are studying whole populations, this does not mean that genetics explains 60 per cent of an individual’s performance, but rather that genetics explains 60 per cent of the differences between individuals, in the population as it exists at the moment.

“This means that heritability is not fixed – if environmental influences change, then the influence of genetics on educational achievement may change too.”

Q: If schools were improved, would the heritability of exam results increase or decrease?

A: That’s a very interesting question. We don’t know. It could be that better schools would have more benefits for people who currently do poorly for genetic reasons, and would reduce heritability; it could be that they would have more benefits for people who currently do poorly for environmental reasons, and would increase heritability; it could be that they would have more benefits for people who currently do well for genetic reasons, and would increase heritability; it could be that they would have more benefits for people who currently do well for environmental reasons, and would decrease heritability. Or it could be that they wouldn’t use exams.

Q: What does this research tell us about the UK’s falling position on the PISA international education comparison? Or is that the fault of Facebook and email, like the UK Schools Minister says?

A:  Well, it’s not genetic.  The genes of the UK population don’t change fast enough. It’s probably not due to Facebook, either, but at least that’s conceivable.

Q: Are the results surprising?

A: Not especially. Similar correlations are seen in IQ test results, where we do know from changes over time that environmental differences can have a large impact. It’s a bit surprising that test scores are slightly more heritable than IQ test results.

Q: Could you give a less controversial example, perhaps something like height?

A: Height is an excellent example.  The fact that siblings have similar heights shows the large genetic component when environments are similar; the fact that most people are taller than their parents or grandparents shows the large environmental component when genetics are similar. If you look in a wealthy Western country, height is about 80% heritable. In medieval Europe it would have been much more sensitive to environment, since the nobility were much healthier and better-nourished than the peasants. And if you mixed together people from medieval Europe and modern Europe, about half the variability would be due to which era they came from.

It’s time for our annual holiday drive-vs-fly post.

Last year I looked at safety: for trips where either flying or driving is feasible, flying is much safer.

This year, let’s look at fuel economy and carbon emissions.  Holiday flights tend to be  full, which makes them more efficient. According to random people on the internet, a Boeing  737 with winglets uses about 3 litres/seat/100km, and according to more reliable sources, Air New Zealand’s smaller turboprop Q300’s will use about 5 litres/seat/100km. (Car and jet fuels are different, but they both have an energy density of about 36MJ/litre and roughly two hydrogens per carbon, so are similar for this purpose.)

There are two further complications: firstly, take-off and landing use more fuel (based on Figure 3 in this report, they add about 250km of effective distance). And secondly, the global warming impact of emissions at high altitude is greater than the same amount of CO2 at ground level, by a factor that is uncertain, but in the range 1.3-2.9.

According the Ministry of Transport national fleet data, the average car in NZ seems to use about 10 litres/100km in real life. In terms of fuel use for long-distance flight, a car with 3-4 people would be comparable to flying on a full 737, and a car with 2 people would be comparable to flying on a full Q300 turboprop. For short NZ distances, the take-off/landing cost adds a good 50%, and the extra impact of high-altitude emissions means that cars win out, though by a relatively small margin if there’s only one person in the car.

A direct comparison like this misses the real emissions problem with planes, though. Hardly anyone drives to the Cook Islands or Australia over Christmas — planes let you travel further.

From Stuff, where GJ Gardner are asking for a GST exemption for their industry

“For example, if the average Kiwi purchases a house worth $600,000, they end up paying $90,000 in GST.”

That’s not a house+land worth $600k, that’s just the value of the new building. In Auckland, the ratio of land to building value averages about 60:40, so you’d expect that to be $1.5 million of real estate. The average Kiwi isn’t quite buying at that level.

At that 60:40 ratio, even if there were no increase in construction costs as a result, the impact of removing GST would be to reduce the price of new homes by about 5%. Auckland prices have increased by twice that much this year already, so it’s not going to solve the problem. In fact, the price reduction would almost certainly be smaller than that — if prices are constrained by available money, and GST is decreased, people will spend some of the savings on more expensive construction.

GJ Gardner are right that the only solution to housing prices is more homes, but in Auckland it’s either going to be necessary to decrease the price of land or reduce the amount of land a home occupies to have a big impact on prices.

This Herald story improves a lot after the first few paragraphs. But it would have to.

Almost three-quarters of the depressed New Zealanders who have gone to Sir John Kirwan’s website depression.org.nz are no longer depressed after finishing the six lessons the site offers.

The story goes on to say that an evaluation of the website found that only 3% of the first 13000 people who registered ended up finishing the six lessons.

Public health officials say the result makes the website, and the $5 million-a-year advertising campaign around it, one of the New Zealand’s most successful public health campaigns.

Based on the 3% figure applied to the current 40000 registrants that doesn’t sound at all plausible: the relevant figure would have to be the 26% who completed at least two lessons, 48% of whom were no longer depressed. That comes to about 5000 people out of 40000. They estimate that 10-15% would have recovered without any intervention, or about 5000 out of 40000.  So, any evidence of public-health benefit needs some information about how representative the 26% were: if they included everyone who was going to recover anyway, there’s no benefit; if they were completely representative, there’s a big benefit. Both extremes are pretty unlikely, but we can’t tell any more from the information in the story.

The evaluation report may have more detail that really does show this website has been effective, but the report doesn’t seem to be public (yet).

[Update: I hadn’t noticed that this had also been nominated for Stat of the Week, with much the same arguments.]