Posts written by Thomas Lumley (2644)

Complicated interventions benefit from pilot studies, where you try to implement the intervention and see how feasible it is.  These are not designed as evaluations of how good the intervention is; they’re typically too small for that and they may have insufficient attention paid to representativeness.  You typically still would look at the outcome of the intervention, and you would have some idea of what you hoped to see.  As Dan Davies says if you don’t make predictions, you won’t know what to be surprised by (and if you don’t make recommendations, you won’t know what to be disappointed by)

In the new young-offenders bootcamp program, there has been a pilot with ten participants.  According to the news, 7 out of 10 have reoffended so far. Since one out of ten died, it would be generous to summarise the proportion with bad outcomes as 8 out of 10.

Speaking to RNZ, acting senior manager in charge Iain Chapman said at the time the pilot began, the 10 participants were the “most serious and persistent young offenders in the country”.

Going into the pilot and expecting no reoffending would have been naive, he said.

This is absolutely true.  What he didn’t say — and should have — was how much reoffending was reasonable to expect. Did he expect better results than two out of ten? Maybe he didn’t. Perhaps one out of ten is what he expected and getting two out of ten is an amazing success. That wasn’t the impression that the government and the media were giving when the program was announced, though. In particular, getting two out of ten not to reoffend doesn’t stack up well against the death.

If the ten pilot participants had been a representative sample of the sort of people who would go into the program, we could do some statistics.   However, we can’t really do this because the pilot program is so small and we don’t know how the participants were chosen. They presumably weren’t chosen specifically because they were unlikely to benefit, but we can’t say much more.

I would have expected that somewhere on a server in Wellington there is a business case for this program that has someone’s best guess at the likely success rate. It would be good to know if that person is surprised, or disappointed.

From economist Justin Wolfers on BlueSky, the record of payroll employment revisions by the US Bureau of Labor Statistics

First: look at 2020!

Next, though, the purple and green lines are quite close together compared to the scale of year-to-year change even when there isn’t a pandemic.

On the other hand, people do actually care about differences of the size we see between the initial and revised estimates, as is demonstrated by the stock market reaction to the revisions.  What this really shows is how difficult the estimation problem is.  People care deeply about changes that are almost invisibly small on this graph, and that are right at the limit of what’s feasible statistically.

The ideal solution is probably for people to be more relaxed about small changes in estimated payroll employment, just as the ideal solution for political opinion polling discourse is for people to have a more realistic view of the limits of estimation. Alternatively, if people want to be unrelaxed about small differences, they need to be willing to pay more to get better estimates.

As you may have heard, President Trump has dismissed the head of the US Bureau of Labor Statistics, claiming that payroll employment figures presented by the BLS were faked to make him look bad.

Politicians meddling with official statistics is a bad idea.  This isn’t because official statistics are Pure and Holy and True and above mere political concerns; it’s because official statistics are messy and difficult and hard to get right, but also very valuable.   The benefit-cost ratio of good official statistics is very high; for the NZ Census the ratio was estimated some years back as 10.  National and local governments, non-profits, and businesses use official statistics to make decisions and the stock market responds to the numbers.  On the other hand, the benefit-cost ratio of bad official statistics is very low — if no-one believes the numbers, there’s not a lot to be gained by publishing them.  Since estimation is messy and difficult and hard to get right, trust in official statistics agencies is critical for trust in official statistics.

Agencies don’t always do it perfectly.  It can definitely be necessary to have some sort of independent review at times.  I was on the External Data Quality Panel looking at the 2018 NZ Census, and there has just been an independent review of the UK Office of National Statistics.  The goal is to make sure the agencies have good procedures, evaluated carefully, to produce the best feasible answers.   Political interference, on the other hand, is discouraged by national and  international principles for official statistics.  It’s hard to get rid of once you have it, and very hard to prove you’ve gotten rid of it — like black mould.

The American Economic Association put out a statement on Friday saying that getting rid of the BLS head this way was  a bad idea.  They don’t do this sort of statement very often.  The International Statistical Institute put out a statement today — they do this more often, but it still takes a fairly significant event to get them moving.  The American Statistical Association haven’t said anything yet, but they were all travelling to their annual conference over the weekend, so it might come soon.

From 2015 in the Herald and in StatsChat

Asthma could be cured within five years after scientists discovered what causes the condition and how to switch it off

As I noted at the time: nope, and nope.

Following up, the drugs in question, called calcilytics, continue to not be used to treat asthma. Hope is not entirely lost — a 2022 research paper says

these data firmly suggest that first-in-human studies will be feasible, desirable and achievable in the short term.

So it might still be true that this research eventually leads to useful treatments, but it certainly didn’t happen five years ago.

From XKCD, the most commonly reported plant and animal on iNaturalist for each US state (click to embiggen)

This is mostly about selection biases of various types: how recognisable the plant or animal is and how interesting.  Saguaro, in Arizona, are not exactly rare, but they aren’t the most commonly seen plant. They are famously associated with the southwest desert and immediately recognisable, so they get reported often.  In other states, the most common plant is genuinely common: yarrow in Montana, California Poppy in California, Amur honeysuckle (sadly) in some Kentucky and Indiana.

So what is it in New Zealand? Looking at the “research-grade” reports, because it’s easier, the most common animal seems to be the kererū, with the pīwakawaka second. Easy to recognise, popular, interesting. For plants, it’s the māhoe, which I wasn’t expecting.

• In “where are they now” followup: a 2016 StatsChat post examining a claim that dementia cures were just five years away. They weren’t.
• Data Strips: A nice lookup at ways to show the distribution of a single numeric variable “in-line”
• As foreshadowed by XKCD, mobile phone acceleration sensors are now genuinely warning of earthquakes-in-progress (media, research paper)
• Stuff reports a reported sighting of the South Island kōkako. From the South Island Kōkako Trust (via Mike Dickison), a map just of probable encounters with South Island kōkako (ie, leaving out their large collection of “possible” encounters). They’re everywhere!
  
  That’s the problem. If they really existed in any significant fraction of the places they’ve been reported, you would expect them to be seen a lot more often including by people with cameras at the ready.  It’s not really feasible for a bird to be on the edge of vanishing, all over the South Island simultaneously, for decades, which is why I think it’s an ex-parrot.

As you probably have heard, tap water in Gore has been running just above the regulatory maximum of 11.3mg/L of nitrate. Nitrate in high enough doses converts hemoglobin to an inactive form, and babies are more susceptible. There are other risks, but they are more speculative according to the CDC. The risk should be pretty low when the concentration is just above the regulatory limit — that’s the point of regulatory limits — and it’s mostly for babies and pregnant women. Still: not ideal, and  you don’t want to let water standards start to slip.

That first link, though, is from 1News.  A couple of months ago they ran a story headed Seven things to eat or avoid to lower your blood pressure. Number 2 on the list is beetroot, based on its high nitrate content. Yes, the same nitrate that’s in the water in Gore. Beetroot juice contains quite a lot more nitrate. A research paper on sports supplements suggests the effective dose would be about 5 mmol per serving, which translates to a bit more than 300 mg, or more than 25 litres of Gore tapwater.

Obviously there’s difference between sports nerds deliberately drinking beetroot for the nitrates and nitrates turning up as an unwanted contaminant for young and old.  The risks are different, and consent matters. Still, it’s a recurrent irritation that the “nitrates good in massive doses” and “nitrates bad even in small doses” stories don’t get cross-referenced a bit more in the media, and we don’t have a more quantitative approach to the risk.  If we’re supposed to believe it’s actually risky to brush your teeth with water that’s 1% above the regulatory limit, the limit is in the wrong place.

• Stuff has Eye-watering new food price stats just revealed: Price of butter has nearly doubled in a year. If you read a bit further it turns out “Butter surged to $8.60 per 500g, a 46.5% annual increase”.  A 50% increase isn’t doubling.  If it had doubled it would be $11.80 per 500g, not $8.60. Those are importantly different numbers.
• The Royal Statistical Society has given out its annual “Statistical Excellence in Journalism” awards: Some of the winners: How many toddlers and babies are living in temporary accommodation in the UK?  and The rise in reported maternal mortality rates in the US is largely due to a change in measurement and an interactive tool linking information about members of the House of Lords, and  about how a crime victimisation figure excludes stalking, sexual assault, and some other crimes
• To add to all the rugby here, Achim Zeileis and colleagues have probabilistic forecasts for the UEFA Women’s Euro 2025 competition

From 1News (from the Conversation): Your mum was right: soup can aid recovery from winter illnesses

My mum was right about many things, but she did not say that soup could aid recovery from winter illnesses.  To feel better with a cold she would recommend hot lemon and honey (for adults, perhaps with a splash of whisky), but she didn’t ascribe any particular therapeutic powers to it.  My preference is Tom Yum soup, again without any claims about faster recovery.

The original story at The Conversation, unlike the 1News piece, actually links to things (The Conversation has mastered the technology of the hyperlink).  One thing it links to is the review article by the author of the story; another is a medical encyclopedia snippet saying soup may make you feel better but won’t affect your recovery.

There’s not as much to the review article as one might hope, even though the researchers seem to have done a very thorough search. People tend not to do high-quality randomised trials of non-standardised home interventions, and they also tend not to do high-quality randomised trials in common cold, so when you combine the two, there’s not a lot to find.  The researchers say they found four studies that looked at symptom duration, one of which showed evidence of a benefit.

I went to look at that one. It’s available from the National Library of Medicine, and it was published in the European Journal of Integrative Medicine.  It does describe a randomised trial, done in Iran.  Let’s look at how the abstract of the paper starts

SARS-CoV-2 causes severe acute respiratory syndrome prompting worldwide demand for new antiviral treatments and supportive care for organ failure caused by this life-threatening virus. This study aimed to help develop a new Traditional Persian Medicine (TPM) -based drug and assess its efficacy and safety in COVID-19 patients with major symptoms.

So it’s not really soup in the usual sense. Some of it sounds quite nice: chicken and barley soup with rosewater and saffron and fig. Other parts maybe not so much: your day started with a tablespoon of herb-Sophia seeds, a relative of cabbage and mustard.  The control group was normal hospital food, so this certainly wasn’t blinded.

In any case, the claim is that this soup has therapeutic effects on Covid-19, and there I’m prepared to be substantially more skeptical than with the common cold.  There wasn’t any sign that the treatment  led to earlier cure or prevented the illness getting worse. The study claims a reduction in four self-assessed symptoms, but the reduction isn’t big and the study decided to use twice the normal threshold for false-positive results — only one of the four reductions would barely meet the usual threshold.   There’s nothing magical about the usual threshold, but there also doesn’t seem to be anything magical about the soup.

If you have Covid, consult your doctor. If you have a cold, your favourite soup might well be a good way to feel better for a bit, especially if someone else makes it for you.

We’ve seen in the past that NZ has very high estimated numbers of homeless people by international standards, and that this is at least in part because we have a very broad definition of “homeless”.

In this podcast, journalist Elizabeth Spiers talks to Brian Goldstone about his new book on homelessness in the US, and in part about how the problem is a lot broader than the official homelessness statistics.  His book takes into account the same sorts of people without a home that the NZ statistics do, and his estimate that the true number is about six times the official number would rate the USA as a little worse than NZ.