Success rates

Complicated interventions benefit from pilot studies, where you try to implement the intervention and see how feasible it is.  These are not designed as evaluations of how good the intervention is; they’re typically too small for that and they may have insufficient attention paid to representativeness.  You typically still would look at the outcome of the intervention, and you would have some idea of what you hoped to see.  As Dan Davies says if you don’t make predictions, you won’t know what to be surprised by (and if you don’t make recommendations, you won’t know what to be disappointed by)

In the new young-offenders bootcamp program, there has been a pilot with ten participants.  According to the news, 7 out of 10 have reoffended so far. Since one out of ten died, it would be generous to summarise the proportion with bad outcomes as 8 out of 10.

Speaking to RNZ, acting senior manager in charge Iain Chapman said at the time the pilot began, the 10 participants were the “most serious and persistent young offenders in the country”.

Going into the pilot and expecting no reoffending would have been naive, he said.

This is absolutely true.  What he didn’t say — and should have — was how much reoffending was reasonable to expect. Did he expect better results than two out of ten? Maybe he didn’t. Perhaps one out of ten is what he expected and getting two out of ten is an amazing success. That wasn’t the impression that the government and the media were giving when the program was announced, though. In particular, getting two out of ten not to reoffend doesn’t stack up well against the death.

If the ten pilot participants had been a representative sample of the sort of people who would go into the program, we could do some statistics.   However, we can’t really do this because the pilot program is so small and we don’t know how the participants were chosen. They presumably weren’t chosen specifically because they were unlikely to benefit, but we can’t say much more.

I would have expected that somewhere on a server in Wellington there is a business case for this program that has someone’s best guess at the likely success rate. It would be good to know if that person is surprised, or disappointed.