Posts written by Thomas Lumley (2645)

• From the UK Office of National Statistics, some fascinating graphs about the age distribution of deaths by suicide and by drug poisoning.  The graphs make the generational differences really clear: it’s my generation in both cases.
• From the comic Saturday Morning Breakfast Cereal, the issue of mathwashing by AI
• From Radio NZ: discussion of Auckland Transport’s new CCTV camera network.
• New York Times on how people perceive their incomes  as high/medium/low
• “Data Visualisation in the Humanities”, from New Left Review. “What interests us is visualization as a practice, in the conviction that practices—what we learn to do by doing, by professional habit, without being fully aware of what we are doing—often have larger theoretical implications than theoretical statements themselves.“
• Why does Google Maps have a fake New York neighbourhood called “Haberman”?

I’d been worrying about this, so it’s an excuse to do some arithmetic in a news setting.

Hannah Martin at Stuff has a story about the current measles outbreak

Of the 516 cases across the country, 299 had not been vaccinated at all.

A further 154 people who contracted measles this year did not know their vaccination status, ESR data showed.

I thought that implied a surprisingly high number of cases in vaccinated people.  Here’s the ESR report. Let’s compare fully-vaccinated and unvaccinated people, and restrict to those over 4 where ‘fully vaccinated’ means two doses of vaccine, not just ‘on schedule so far’.

There were 30 cases in fully vaccinated people, and about 150 in unvaccinated people.  What sort of ratio would we expect?  Roughly 90% of people have been vaccinated, so if the vaccine had no effect and exposure was uncorrelated with vaccination we’d be expecting about a 10:1 ratio in favour of vaccinated cases. We get a 1:5 ratio the other way.  This very crude comparison suggests about a 50-fold reduction in risk from full vaccination, which is about what’s expected.

It’s a bit more complicated than that. Firstly, the vaccination coverage figures don’t count immigrants.  People who immigrated as kids from somewhere that needs a visa would typically have been vaccinated.  It’s less clear for adults — I had one dose of measles vaccine as a baby  and one when I started my PhD [edit: and another one when I applied for US residency], but no-one asked when I moved here. If immigrants are less likely to be vaccinated, the 10:1 population ratio is smaller and we’d expect more unvaccinated cases.

Second, people who aren’t vaccinated are more likely to be exposed to measles, because of the way vaccination is distributed in the population.  That’s true both for ‘vaccine hesistant’ groups and (as Kirsty Johnston and Chris Knox report) because of poverty and access limitations. If you aren’t vaccinated, it’s more likely that your friends and neighbours aren’t.  Again, this would give us more unvaccinated cases.

Putting these together, the proportion of unvaccinated cases might be a bit lower than we might expect, but not seriously lower.  If there is a discrepancy, it could be due to trusting self-report of vaccination status — adults who know they got all the vaccines that were on offer as kids might well assume they were fully vaccinated against measles if they didn’t have the paperwork to check.

If you’ve only had one dose of the measles vaccine, or if you aren’t sure, you need another one.  Check with your doctor about the current recommendations — and if they’re short of vaccine now, make a note to check again in a few months.  We eradicated the ancestor of measles in 2011, the cattle disease rinderpest, but we’re not close to eradicating measles.

There’s a pretty good piece on Stuff about bias in the justice system that might be attributable to biased algorithms. You should read it.

The story talks about two specific people, one who had a low predicted risk and did re-offend, and one who had a high predicted risk and, well, we don’t know yet.  That’s evidence that the model isn’t perfect; it doesn’t tell us much about how good or bad it is: if you have a well-calibrated model and it says someone has a 0.06 chance of re-offending, then out of every sixteen people like that you’d expect one to re-offend.  Individual cases aren’t very helpful in assessing how good or bad the system is; you need statistics.

As the story makes clear, though, if you want a system that gives Māori and Pākehā the same sentences, simply leaving out the ethnicity variable from your model isn’t going to do that.  Differences by ethnicity are all over the data. A statistical model is going to see who is in prison, and send along more people like that.

Part of the problem (as the story says) is the data: we don’t actually have data on re-offending, only on re-conviction, and the difference between the two involves the justice system and its biases, and there’s a potentially very nasty feedback loop there. But that’s only part of the problem.  The other part is that basing imprisonment on the likelihood of re-offending is going to result in longer terms in prison for people from groups that re-offend more often. And that will include Māori: the over-representation of Māori in the prison population is not just because the justice system is racist, but also because society is racist.

There’s not just a problem with the answer that the model gives; I think there’s a problem with the question, too. The intuition behind predictive sentencing is that if you have two people convicted for the same crime, and they are otherwise similar, and one of them is more likely to commit future crimes, you want to keep that one out of the community for longer.  For me, at least, the intuition relies quite strongly on the ‘otherwise similar’ qualification.  If you came along and said “young people are more likely to commit future crimes than old people, so we should lock them up for longer”, I wouldn’t be at all persuaded. The same for poor vs rich. Or men vs women. Or Māori and Pākehā.  These don’t seem like the sort of relevantly-similar-but-different-risk distinctions that are intuitively a good idea to base imprisonment on.

That is, I think one of the reasons many people don’t like the outputs of predictive sentencing models is that we don’t actually believe in sentencing based on risk of re-offending; at most, we believe in something much more complicated that the models don’t try to do.

I have to admit a distinction here between initial sentencing and parole. Parole decisions, according to the Parole Act must consider both the likelihood of further offending; and the nature and seriousness of any likely subsequent offending. Parole fundamentally does involve the risk of re-offending. Initial sentencing has a lot of purposes, and risk of re-offending is much less tightly linked with it.   According to the story, though, the predictive model is an input to both processes, and similar models are certainly an input to sentencing in the USA.

And finally, it’s important to remember that one of the original reasons people built statistical models to help with sentencing and parole decisions was that it was previously being done by the humans who are the source of the biased data we’re complaining about. Getting rid of statistical models and just relying on the fairness and objectivity of people in the justice system isn’t a panacea either.

NewsHub and Radio NZ had stories about New Zealand blackcurrant juice and its claimed ability to make exercise less tiring

Scientists at Plant & Food Research have found that juice from New Zealand blackcurrants consumed prior to exercising could increase motivation to adhere to exercise.

and

Plant and Food science group leader, Dr Roger Hurst says New Zealand blackcurrants are high in the properties, which help your body recover from exercise. He also said polyphenol lifted mood, so that people felt more inclined to exercise better and for longer.

The open-access research paper is here, and the press release is here. It was a placebo-controlled randomised experiment, which is good. Even better, it was pre-registered at the ANZ Clinical Trials Registry, so we can tell what the researchers said they planned to do and match that up to what they say they have done.

Participants drank real or fake blackcurrant juice before a long period of gentle walking on a treadmill.  People who drank real blackcurrant juice felt better (by about half a point on a ten-point mood scale) and felt they were exerting themselves less (by about one point on a twenty-point scale). That’s all as the stories claim.  The researchers also found a bunch of biochemical things that were consistent with what they expected.

The main critical thing I’d say is that the study registration listed the effect on exercise duration as the first primary endpoint; that is, the main question the study was about.  There was no real evidence of any effect on exercise duration, and this negative finding for the primary study question has been rather deemphasised in the press release and media coverage. The research paper does give the exercise results, but doesn’t mention their status as a primary endpoint — I wouldn’t have known if I hadn’t looked up the study pre-registration, and I might well have assumed this was a small preliminary study that wasn’t really expecting to find exercise differences.

Which, of course, is why pre-registration is so helpful.

I make a lot of fuss about the media not linking to scientific papers. So do many other people who write about science.  This isn’t just a weird kink: part of what makes it science is that you can find out what the researchers did and why.

Usually the lack of links is because the journalist doesn’t put them in.  Today, we have two other cases, in sort of opposite directions.

First, the world’s largest ex-parrot. According to a lot of new stories, especially in the UK, there is a newly discovered giant dead parrot from central Otago. It stood a metre high. Some of the more excitable stories claim it was a cannibal, but that seems to be because they have ‘cannibal’ and ‘carnivore’ (or ‘omnivore’) confused.   Many of the stories give a link to a research paper in the Royal Society’s journal Biological Letters. The link doesn’t work. The most likely explanation is that the journal just hasn’t got around to publishing the paper yet, and the link will start working in a day or two.  But it’s still not good science practice or science publishing to deliberately work up news stories about research that isn’t available to read.

Second, there’s a story on Stuff about a company claiming to use epigenetics to personalise health advice.  I say ‘claiming’ because (a) the company doesn’t actually measure any epigenetics, and (b) there isn’t any evidence that I (or the outside expert quoted in the story) know of that says it works to customise health advice using epigenetics.   The health advice is quite likely perfectly sensible and beneficial to its customers. And if a company wants to use head measurements as well as height and weight to say how you should exercise, there’s no real reason they can’t.  It would have been nice to see some question raised on how epigenetics gets involved, though. Here, the lack of any suggestion of links is what raises questions — not so much about the story as about what the company is actually doing.

I wrote a post last year on why one might or might not care about ‘business confidence’: basically, it’s not of much real interest in itself, but it may be a leading indicator, a useful predictor of short-term changes in employment or investment.  As I said then “Whether a business confidence survey will be economically useful is an empirical question, to be answered by data.”

In light of the new fall in the business confidence survey, it’s worth noting that this question has been answered by data. The answer is “No”.

David Hood has looked at whether NZ business confidence predicts next-quarter GDP. There’s a weak negative correlation: if business is confident, there’s a slight tendency for the economy to get worse in the next quarter. More dramatically, he finds that business confidence from countries other than NZ is a better predictor of NZ GDP than NZ business confidence is.  If you want to know how the NZ economy is doing, it seems you’d be better off asking Chinese or Swedish or Canadian businesses than Kiwi businesses.

One big component of the problem is that business confidence in NZ has been lower when we have a left-wing government led by a female PM. Since the economy has done better  when we have a left-wing government led by a female PM, we get a negative correlation.  That’s not to say that Labour women necessarily run the economy better; most of the variation isn’t something that the government has much control over; it’s not that Helen Clark was holding off the Global Financial Crisis and electing John Key then let it happen.  Still, when business confidence is affected by that sort of bias it’s not going to be as good an economic indicator, and even when you try to subtract off the bias, there doesn’t seem to be much information in the NZ business confidence index. 

You  might ask why I’m believing some random person on the internet rather than the ANZ survey.  It’s not a matter of reputation — we have the analysis, and StatsNZ has the data. Business confidence surveys can, in some settings, be a useful economic leading indicator.  In New Zealand, they haven’t been.

There’s a story at the  Huffington Post, based on a new research paper, saying

Genes account for about 80% of a child’s risk of developing autism, a massive new study finds.

That link is not from the HuffPost story (they don’t link), but via Spectrum News, whose story is similar except they actually use the word “heritability” in a few places.

The abstract of the research paper says

Based on population data from 5 countries, the heritability of ASD was estimated to be approximately 80%, indicating that the variation in ASD occurrence in the population is mostly owing to inherited genetic influences, with no support for contribution from maternal effects.

so this is a case where the claims come from the scientists — but where the claims aren’t as strong before you translate them from Science to English.

If you think about it for a few minutes, it obviously can’t be true in a straightforward sense that 80% of the risk of developing autism is genetic(or, more precisely of being diagnosed with it, since that’s what can be measured).  Autism (diagnosis) is much more common than it used to be. For example, the US Centers for Disease Control report that from 2000 to 2014, “ASD prevalence estimates increased from 6.7 to 16.8 per 1,000 children aged 8 years, an increase of approximately 150%.” Absolutely none of this change is genetic: fourteen years isn’t long enough for population genetic changes. In the US, over that period, something non-genetic that varies with time was responsible for the majority of the autism (diagnosis) risk.  Improvements in diagnosis are probably one contributing factor.

Heritability is an important technical measure that sounds more interesting than it actually is.  It’s related to whether things are genetically determined, but it isn’t the same. For example, in humans, “number of legs” has very low heritability, but the reason nearly everyone has two legs is genetic.

More seriously, suppose that a condition requires both that you have a particular genetic variant and that you experience a particular environmental exposure.  Whether the condition ‘looks’ genetic or environmental will depend on who you compare.  If you compare among people of northern European ancestry living in NZ, melanoma skin cancer looks to be mostly environmental: it’s caused by sunburn.  If you compare among people living in South Africa, melanoma looks substantially genetic: it’s caused by a lack of melanin in the skin, and the genetic contributions to skin colour are moderately well understood.

• AI Ethics: Seven Traps, from freedom-to-tinker
• The NZ$ Herald has a front page story about over-screening for eye conditions, which uses the wonderful word “incidentaloma” (an unnecessary and accidental diagnostic finding)
• On the avocado-toast libel of millennials: actually millennials earn more money, and spend less on food and entertainment than the baby boomers did at the same age (inflation-adjusted)
• Lisa Charlotte Rost looks at an Economist graphic about unprecedented Arctic ice melting and redraws it so it’s true.
• In a new podcast by The Spinoff, Russell Brown talks to two AI/algorithm experts about what ‘algorithms’ are.
• A widely-publicised study of gut bacteria and autism in mice has problems. I’m quoted not believing the statistical analysis.
• From the Sydney Morning Herald, a graphic depicting the size of a blue whale in terms of more familiar, everyday objects (though the typical adult blue whale is a few metres shorter than that)
  

From Sky News and ABC News and the Guardian

Four out of five people identified by the Metropolitan Police’s facial recognition technology as possible suspects are innocent, according to an independent report.

The Police prefer a different definition of accuracy that says the error rate is about 1 in 1000.  It’s not surprising they prefer to say the error rate is 1 in 1000, but you might wonder how you can get two definitions that different.

The full report is here (PDF). In a set of trials in UK cities, the system identified 42 matches to people on the watchlists.  Of these, 8 were confirmed as correct, 16 were identified as wrong just by looking at the pictures, 14 were identified as wrong by checking id, and for 4 of the matches the police couldn’t find the person to check their id.  If you’re feeling really generous you could say the police would be just as good at discarding poor matches in real life as they are in a carefully audited field trial, and you might then say you had 8 right, 14 wrong, and 4 “don’t know” in cases where the police were convinced enough to go up to someone and ask for id; that’s still not 50%.

The Police definition of error rate is the number of errors as a fraction of all the faces scanned.  If you make 14 or 18 or 34 errors in scanning tens of thousands of faces, the error rate per face scanned will be low.  The problem is that, under this definition, the error rate of just not using the facial recognition software is even lower than the 1 in 1000 from using it.

What the stories don’t really do is ask what the error rate should be? The right answer would need to combine the harm done by false matches and the benefit from true matches.  One might also want to consider the benefits from deterring crime or the harm from giving the police more pretexts to challenge people they didn’t like.   In medicine we tolerate screening tests that have error rates worse than this facial recognition system — but in situations where people give consent to be screened and to any further follow-up.

It’s hard to answer the question of what error rate would be ok, but it’s important to ask it.

The NZ Police have put out a report “based on the most frequent data requests made by members of the public under the Official Information Act and is organised according to the largest areas of demand that Police responds to.” (via Nat Dudley on Twitter)

That’s great. There are numbers and tables and graphical summaries.  Very useful.

On the other hand, this is typical of the graphs

The report has a lot of charts like this, showing trends over time.  They look as if they are using areas to depict the data, but they all have truncated y-axes that make the year-to-year variation look more dramatic.  The bottom 75% of this chart has gone missing.

If you want to use a truncated y-axis you can, with a line chart and axis designs that don’t make the bottom of the graph look as if it’s zero.  For example, patterns in the Keeling Curve of atmospheric CO₂ are more apparent if the whole range down to zero ppm isn’t shown. In this case, though, there’s no obvious benefit. It’s just misleading.