Stats Chat

If you’ve been paying attention to the news you might have read that Forbes ranked NZ second in the world for COVID safety. If you chose more careful news sources, that you might instead have read that Forbes published a story about COVID safety ranking constructed by a company called Deep Knowledge Group.  It’s their second try; they published the first round in June.  And, of course, you might remember stories from the Before Times about New Zealand’s  poor ranking on the Global Health Security Index.

So, which of these indexes are right and which ones are wrong? It turns out that’s the wrong question, just as it is about rankings  of the most liveable city. These rankings aren’t trying to be predictive in a way that makes ‘right’ and ‘wrong’ objectively assessable.  What happens is that a group of people gathers together a whole lot of measurements.  These measurements range from ‘good’ to ‘bad’ on some scale. Points get awarded for how close you are to ‘good’ on each measurement and then added up. The experts then look at the rankings they got out, and probably adjust the points a bit to  make them look more plausible — there’s nothing wrong with this; the experts’ judgment is where there’s potentially value added by having this index.

There is a technical problem with reducing things to a sum of numbers in this way — and I say that as someone who likes numbers. We might all agree that, say, willingness to follow mask recommendations is positive, and that having a policy of mask recommendations when appropriate is positive. But these don’t add up: you need both of them.   Being more willing to follow government advice to wear masks does no good if the government is not giving that advice; having the government give the advice does no good if it isn’t followed.  You can’t represent A AND B as a weighted sum.  Weighted sums are a very limited subset of ways of combining ordered scales.

It’s clear how much opinion goes into the index construction when you see that the Global Health Security Index and the new COVID index share only two countries in their top ten: Australia and South Korea.  The United States and the UK top the GHS Index; Germany and NZ top the Deep one.  The real question is whether the indexes are useful, and they probably are — or, more precisely, whether the reports surrounding the indexes are useful, where the experts talk about the ways countries differ and their strengths and weaknesses.

If you wanted to think of the indexes as predictive, you’d want to look at the June edition and see how it’s fared.  Most of the top ten have done ok, but Israel, at number 3,  has had a bad few months, and now has 27,000 active cases in a country with less than twice the population of New Zealand.  They have been demoted to number 11: four places ahead of Taiwan, 21 ahead of Vietnam, and 83 ahead of Mongolia.

The index and its surrounding report contain useful information; for example they identify (admittedly with 2020 hindsight) the importance of political will to act and social acceptance of restrictions.  The ranking of NZ as the second-safest country is only meaningful if you don’t interpret ‘safest’ in some crudely reductive way as referring to the likely number of cases or hospitalisations or deaths.

There are about forty COVID vaccine candidates now, with several heading to Phase 3 trials.   The  CDC has suggested there could be a vaccine before the US presidential election. How could that happen? There’s one obvious answer, sadly, but how else could it happen?

The FDA has said it would expect that a COVID-19 vaccine would prevent disease or decrease its severity in at least 50% of people who are vaccinated.  In the current circumstances, if your vaccine was that effective it would be false economy to run a trial that was too small — you’d want to be very confident of a positive result if your vaccine met the mark.

You might design a trial that ran until you had seen, say, 150 infections or symptomatic cases or whatever. If you ended up with a split of 50 vs 100, you’d estimate a 50% reduction with the vaccine, and the uncertainty interval (‘margin of error’) in would stretch to a 30% reduction, which correspond to the FDA’s guidance.   Only 150 cases perhaps doesn’t sound like much, but remember that at a very high infection rate of 1% you’d need 10,000 people in the control group and the same number in the treatment group. That’s a big trial.

It’s possible, though, that your vaccine is much better than just a 50% reduction in cases. If it actually prevents 90% of cases, you could have run a much smaller trial, at lower cost, finished earlier, and saved more lives.  On the other hand, if you ran the smaller trial and the effectiveness is only 50%, you are in big trouble — you won’t get approved, but you’ll find it harder to run another trial.

So, what we do is to set up a big trial that can detect a 50% benefit, and peek at the data part of the way through.  If the vaccine turns out to be 90% effective, you’ll be able to stop earlier than planned and declare victory.  You’ll still have recruited more people than if you were betting everything on 90% effectiveness, but you won’t be betting everything on 90% effectiveness.

Stopping early requires strong evidence.  It requires strong evidence because it’s easy to get fooled by randomness in early trends. It requires strong evidence because early stopping means less data on safety. And it  requires strong evidence because people are going to suspect political interference.  One popular guideline for designing early-stopping rules, to give us something concrete to look at, says that you take margin you’d need to be convinced at the end of the trial, and use that margin earlier on.  In our hypothetical example, we had 50 cases difference as the success criterion at the end of the trial, so you’d look for 50 cases difference at earlier timepoints as well.  That’s a much bigger relative difference early on:  if you only had 100 cases, a difference of 50 is 25 vs 75; with only 60 cases, it’s 5 vs 55.

My understanding is that a 90%-effective vaccine would be surprising, but not inconceivable and a 70%-effective vaccine isn’t at all unlikely, so interim analysis and the potential for early stopping do make sense.  Political interference doesn’t help, though: even more than most drugs, a vaccine only counts as effective if lots of people are willing to take it, and a trial that doesn’t engender that willingness is a failure no matter the statistical results.

• A surprising (to me) statistic from the Washington Post: 100 US law enforcement officers have died of COVID contracted at work. That’s more than all other line-of-duty deaths this year put together. And it  doesn’t include 150 more deaths that the Officer Down Memorial Page is still reviewing.
• A particularly malign case of ‘predictive policing’ in Florida.
• You’ve probably heard about the UK’s  plan to impute A-level grades after cancelling exams. A good discussion of why this is a hard problem and how to avoid solving it. 
• “Sooner or later we are going to have to have an inquiry into the role of statisticians in the Covid-19 crisis,” declared the journalist Ross Clark in the Daily Telegraph recently.” David Spiegelhalter would relish the opportunity.
• Alert-level decision in NZ are hard; getting them wrong either way can have very bad consequences. Ars Technica has an excellent report on a similar sort of decision: whether or not to evacuate Houston as Hurricane Laura approached.
• The FDA Commissioner, talking about plasma from recovered COVID patients as a treatment, managed to get both the estimated benefit and the reliability of the evidence badly wrong. Hilda Bastian writes about the implications for informed consent by future patients

There’s an interesting poll in the NZ$ Herald asking whether the four-day extension to the Auckland level 3 ‘lockdown-lite’ was appropriate or not. Here’s a graph of the regional results (excluding the worryingly-small fration of “don’t knows”). The purples are ‘yes, should have been extended’, with the light purple being ‘the four day extension was appropriate’ and the dark being ‘longer would have been better’. The oranges are ‘no, should not’, with the light orange being ‘should not have been extended’ and the dark being ‘should not have happened at all’

As you can see, there’s pretty strong consensus across the country.  You’d expect people outside Auckland, who get the benefits but with less of the cost, to want tighter restrictions, and the patterns seem to fit that.

The Herald also explores differences by age, gender, and income. It’s hard to say anything too strong about many of the differences, because we’re only quoted a margin of error for the whole survey, not for any of the subgroups.  In some cases I can work it out: if Auckland and Canterbury were represented in the sample in proportion to their actual population sizes, the difference between the purple bars would be right about at the margin of error.  In general, though, the overall margin of error is pretty useless for most of what makes the story interesting.

Oh, and also.

Auckland is strongly divided over whether extending the lockdown was an appropriate response to the resurgence of Covid-19, a new poll shows.

But the exclusive new poll shows rest of the country was far more accepting of the Super City being kept in alert level 3 for almost three weeks – with many wanting it extended even longer.

Emphasis added.

No. That’s really not what the poll says

Team Ratings for Round 10

As an indication of the care needed at present in determining home ground advantage, the comprehensive scores site flashscore.com.au still has the Rebels v Waratahs game as being played in Melbourne. (Click on the match result to see the scorers and times and match details at the bottom.) Fortunately I checked beforehand that it was to be played in Sydney so I gave the Waratahs home ground advantage and predicted them to win. Otherwise I would have predicted the Rebels, even if played on a neutral ground.

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 18 matches played, 15 of which were correctly predicted, a success rate of 83.3%.
Here are the predictions for last week’s games.

Predictions for Round 10

Here are the predictions for Round 10. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 18

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 102 matches played, 66 of which were correctly predicted, a success rate of 64.7%.
Here are the predictions for last week’s games.

Predictions for Round 18

Here are the predictions for Round 18. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for the Semi-finals

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 101 matches played, 78 of which were correctly predicted, a success rate of 77.2%.
Here are the predictions for last week’s games.

Predictions for the Semi-finals

Here are the predictions for the Semi-finals. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 17

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 128 matches played, 88 of which were correctly predicted, a success rate of 68.8%.
Here are the predictions for last week’s games.

Predictions for Round 17

Here are the predictions for Round 17. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 17

I got caught out a bit this round with games being played earlier in the week and then again on the weekend. As usual my predictions are what they would have been before the games were played, the predictions are produced by running my algorithm over the past data and information about home grounds.

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 96 matches played, 61 of which were correctly predicted, a success rate of 63.5%.
Here are the predictions for last week’s games.

Predictions for Round 17

Here are the predictions for Round 17. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Auckland is going back to level 2 (God willing and the creek don’t rise) on Monday.  Together with my local circle of health and stats nerds, I’m viewing this with some concern. It hasn’t been very long since we had a case show up with no previously known contact to the cluster. Like, Tuesday.  It’s quite possible there are still a few other people in the cluster who haven’t been found yet.

The concern is not that it will actually be dangerous on Monday to be going to work or going shopping. The number of undetected cases will be small; your chance of getting infected on Monday is tiny.  The problem is, as long as there are undetected, infectious cases, your chance of getting infected on Tuesday is very slightly higher. And slightly higher again on Wednesday, and so on in exponential increase. Eventually, it may get dangerous, and  stopping it then is much more costly in health and freedom and money.  In Victoria today they are talking  about the psychological boost of getting a day with less than 100 new cases. If we’re careful and lucky, the current testing and tracing will be enough to stop this cluster exploding; if we’re not, maybe not.

One challenge in COVID risk communication is that the risk is longer-term and social, not immediate and individual. It’s important that as many  people as possible take precautions against spreading the virus: distancing, masks, getting tested if you have symptoms, working from home if that’s feasible, avoiding places with poor ventilation. But it’s not so much important for your or your family’s immediate safety: the risk is currently very low. If someone jogs past you at close range without a mask or stands a bit too close in  a supermarket line, you don’t need to panic — you almost certainly won’t catch the coronavirus. On the other hand, the more people behave that way, the more chance that they outbreak will slowly get out of control.  The appropriate level of care is a lot higher than the appropriate level of fear.

Ideally, everyone would be as careful as if the virus was everywhere, but nowhere near as scared as if it was everywhere. That’s a very difficult balance, and a reason for the ‘be kind’  message when it doesn’t quite work out.