There’s a pretty good piece on Stuff about bias in the justice system that might be attributable to biased algorithms. You should read it.
The story talks about two specific people, one who had a low predicted risk and did re-offend, and one who had a high predicted risk and, well, we don’t know yet. That’s evidence that the model isn’t perfect; it doesn’t tell us much about how good or bad it is: if you have a well-calibrated model and it says someone has a 0.06 chance of re-offending, then out of every sixteen people like that you’d expect one to re-offend. Individual cases aren’t very helpful in assessing how good or bad the system is; you need statistics.
As the story makes clear, though, if you want a system that gives Māori and Pākehā the same sentences, simply leaving out the ethnicity variable from your model isn’t going to do that. Differences by ethnicity are all over the data. A statistical model is going to see who is in prison, and send along more people like that.
Part of the problem (as the story says) is the data: we don’t actually have data on re-offending, only on re-conviction, and the difference between the two involves the justice system and its biases, and there’s a potentially very nasty feedback loop there. But that’s only part of the problem. The other part is that basing imprisonment on the likelihood of re-offending is going to result in longer terms in prison for people from groups that re-offend more often. And that will include Māori: the over-representation of Māori in the prison population is not just because the justice system is racist, but also because society is racist.
There’s not just a problem with the answer that the model gives; I think there’s a problem with the question, too. The intuition behind predictive sentencing is that if you have two people convicted for the same crime, and they are otherwise similar, and one of them is more likely to commit future crimes, you want to keep that one out of the community for longer. For me, at least, the intuition relies quite strongly on the ‘otherwise similar’ qualification. If you came along and said “young people are more likely to commit future crimes than old people, so we should lock them up for longer”, I wouldn’t be at all persuaded. The same for poor vs rich. Or men vs women. Or Māori and Pākehā. These don’t seem like the sort of relevantly-similar-but-different-risk distinctions that are intuitively a good idea to base imprisonment on.
That is, I think one of the reasons many people don’t like the outputs of predictive sentencing models is that we don’t actually believe in sentencing based on risk of re-offending; at most, we believe in something much more complicated that the models don’t try to do.
I have to admit a distinction here between initial sentencing and parole. Parole decisions, according to the Parole Act must consider both the likelihood of further offending; and the nature and seriousness of any likely subsequent offending. Parole fundamentally does involve the risk of re-offending. Initial sentencing has a lot of purposes, and risk of re-offending is much less tightly linked with it. According to the story, though, the predictive model is an input to both processes, and similar models are certainly an input to sentencing in the USA.
And finally, it’s important to remember that one of the original reasons people built statistical models to help with sentencing and parole decisions was that it was previously being done by the humans who are the source of the biased data we’re complaining about. Getting rid of statistical models and just relying on the fairness and objectivity of people in the justice system isn’t a panacea either.