Privacy and mathwashing

The Herald has a story from the Washington Post on an “AI” screening tool for babysitters, that allegedly uses both computer vision and text processing to screen social media for risk factors.   Here are three quotes from it:

1. A company co-founder says

Parents, he said, should see the ratings as a companion that “may or may not reflect the sitter’s actual attributes.”

But the danger of hiring a problematic or violent babysitter, he added, makes the AI a necessary tool for any parent hoping to keep his or her child safe.

The first thing to note about this is that you could make the same claims about astrology or handwriting analysis or a tarot reading. There’s no quantitative information about accuracy given, and it’s hard to see how the company could even know much how accurate its ratings were, or how biased. It’s not even for sure that the risk rating is positively correlated with risk to kids; the company seems careful not to make even a claim this weak.

2.

Parents could, presumably, look at their sitters’ public social media accounts themselves. But the computer-generated reports promise an in-depth inspection of years of online activity, boiled down to a single digit: an intoxicatingly simple solution to an impractical task.

If the algorithms actually predicted risk better and were less biased than typical employers there might be an advantage to this: your social media would be shared with a faceless US company rather than your potential employer, so there might be less actual privacy invasion — after all, some faceless US companies already have your social media. It might be less embarrassing than your boss knowing what your favourite member of the appropriate sex calls you. The computer could also be set up to ignore irrelevant information like whether you talk about your sexual orientation online.   With the setup as it is, that’s not the case, and one of the biggest risks is the completely unfounded appearance of both accuracy and objectivity — “mathwashing” as the jargon puts it.

3.

Where she lives, “100 per cent of the parents are going to want to use this,” she added. “We all want the perfect babysitter.”

One of the significant risks of automating human judgement is that it can go viral. There’s a limit to how well ordinary human prejudices can scale — you’ve got some chance of finding someone who has different biases. The prejudices of one computer checklist, though, can keep someone completely out of an employment sector.