Posts filed under General (3148)

Team Ratings for Round 20

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 114 matches played, 76 of which were correctly predicted, a success rate of 66.7%.
Here are the predictions for last week’s games.

Predictions for Round 20

Here are the predictions for Round 20. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

One of the COVID vaccine trials sponsored by AstraZeneca has been ‘put on hold’ because of  a ‘suspected serious adverse reaction’.  What does that mean and how bad is it?

Some terminology

• Adverse Event (AE): a bad thing that happens to someone in a trial. These are not necessarily related at all to the trial treatment; something counts as an AE even if you’re getting placebo. They range from minor to serious
• Serious Adverse Event (SAE): the adverse event is hospital bad.  More precisely,  fatal, life-threatening, needing hospitalisation, or leading to significant permanent effect, or  could have gotten that way if it wasn’t stopped by treatment.
• Adverse Reaction (AR): an adverse event that was caused by the trial treatment. Some of the COVID vaccine candidates have quite a  lot of these; fever, for example.
• Serious Adverse Reaction: an SAE that was caused by the trial treatment
• Suspected Serious Adverse Reaction: an SAE that might have been caused by the trial treatment
• Suspected Unexpected Serious Adverse Reaction (SUSAR): an SAE that might have been caused by the trial treatment, and wasn’t disclosed to the participant as a reasonably expected result. (Some treatments do have expected serious adverse reactions — some cancer chemotherapy, for example)

It’s standard in clinical trials that you consider SSARs/SUSARs urgently, and for completely new treatments the consideration should be especially careful.  It might turn out on investigation that the event is unlikely to be caused by the treatment (eg, because of timing, or because you find a more likely cause); it might turn out that it is likely to be caused by treatment;  or everything might just be messily inconclusive.

If you have tens of thousands of people in vaccine trials then adverse events will occur, and some of the adverse events will be serious. That’s inevitable. If you have a treatment that could potentially cause a very wide range of adverse events, then some of the serious adverse events will inevitably look like they could have been caused by the treatment.  And, sometimes, they will have been.  Vaccines aren’t intrinsically safe; they are safe when approved because they get thoroughly investigated first.

The immediate question for the trial sponsor and investigators in this case is whether the adverse event, whatever it is, significantly changes the risk:benefit balance of the vaccine candidate, as explained to the study participants (and to the company’s investors).  So far, no-one knows, but AstraZeneca seem to be handling this appropriately, and there’s no obvious reason to expect that to change.

Team Ratings for the Qualifying Final

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 20 matches played, 17 of which were correctly predicted, a success rate of 85%.
Here are the predictions for last week’s games.

Predictions for the Qualifying Final

Here are the predictions for the Qualifying Final. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 19

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 108 matches played, 70 of which were correctly predicted, a success rate of 64.8%.
Here are the predictions for last week’s games.

Predictions for Round 19

Here are the predictions for Round 19. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for the Final

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 103 matches played, 79 of which were correctly predicted, a success rate of 76.7%.
Here are the predictions for last week’s games.

Predictions for the Final

Here are the predictions for the Final. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 18

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 136 matches played, 92 of which were correctly predicted, a success rate of 67.6%.
Here are the predictions for last week’s games.

Predictions for Round 18

Here are the predictions for Round 18. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 1

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Predictions for Round 1

Here are the predictions for Round 1. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

If you’ve been paying attention to the news you might have read that Forbes ranked NZ second in the world for COVID safety. If you chose more careful news sources, that you might instead have read that Forbes published a story about COVID safety ranking constructed by a company called Deep Knowledge Group.  It’s their second try; they published the first round in June.  And, of course, you might remember stories from the Before Times about New Zealand’s  poor ranking on the Global Health Security Index.

So, which of these indexes are right and which ones are wrong? It turns out that’s the wrong question, just as it is about rankings  of the most liveable city. These rankings aren’t trying to be predictive in a way that makes ‘right’ and ‘wrong’ objectively assessable.  What happens is that a group of people gathers together a whole lot of measurements.  These measurements range from ‘good’ to ‘bad’ on some scale. Points get awarded for how close you are to ‘good’ on each measurement and then added up. The experts then look at the rankings they got out, and probably adjust the points a bit to  make them look more plausible — there’s nothing wrong with this; the experts’ judgment is where there’s potentially value added by having this index.

There is a technical problem with reducing things to a sum of numbers in this way — and I say that as someone who likes numbers. We might all agree that, say, willingness to follow mask recommendations is positive, and that having a policy of mask recommendations when appropriate is positive. But these don’t add up: you need both of them.   Being more willing to follow government advice to wear masks does no good if the government is not giving that advice; having the government give the advice does no good if it isn’t followed.  You can’t represent A AND B as a weighted sum.  Weighted sums are a very limited subset of ways of combining ordered scales.

It’s clear how much opinion goes into the index construction when you see that the Global Health Security Index and the new COVID index share only two countries in their top ten: Australia and South Korea.  The United States and the UK top the GHS Index; Germany and NZ top the Deep one.  The real question is whether the indexes are useful, and they probably are — or, more precisely, whether the reports surrounding the indexes are useful, where the experts talk about the ways countries differ and their strengths and weaknesses.

If you wanted to think of the indexes as predictive, you’d want to look at the June edition and see how it’s fared.  Most of the top ten have done ok, but Israel, at number 3,  has had a bad few months, and now has 27,000 active cases in a country with less than twice the population of New Zealand.  They have been demoted to number 11: four places ahead of Taiwan, 21 ahead of Vietnam, and 83 ahead of Mongolia.

The index and its surrounding report contain useful information; for example they identify (admittedly with 2020 hindsight) the importance of political will to act and social acceptance of restrictions.  The ranking of NZ as the second-safest country is only meaningful if you don’t interpret ‘safest’ in some crudely reductive way as referring to the likely number of cases or hospitalisations or deaths.

There are about forty COVID vaccine candidates now, with several heading to Phase 3 trials.   The  CDC has suggested there could be a vaccine before the US presidential election. How could that happen? There’s one obvious answer, sadly, but how else could it happen?

The FDA has said it would expect that a COVID-19 vaccine would prevent disease or decrease its severity in at least 50% of people who are vaccinated.  In the current circumstances, if your vaccine was that effective it would be false economy to run a trial that was too small — you’d want to be very confident of a positive result if your vaccine met the mark.

You might design a trial that ran until you had seen, say, 150 infections or symptomatic cases or whatever. If you ended up with a split of 50 vs 100, you’d estimate a 50% reduction with the vaccine, and the uncertainty interval (‘margin of error’) in would stretch to a 30% reduction, which correspond to the FDA’s guidance.   Only 150 cases perhaps doesn’t sound like much, but remember that at a very high infection rate of 1% you’d need 10,000 people in the control group and the same number in the treatment group. That’s a big trial.

It’s possible, though, that your vaccine is much better than just a 50% reduction in cases. If it actually prevents 90% of cases, you could have run a much smaller trial, at lower cost, finished earlier, and saved more lives.  On the other hand, if you ran the smaller trial and the effectiveness is only 50%, you are in big trouble — you won’t get approved, but you’ll find it harder to run another trial.

So, what we do is to set up a big trial that can detect a 50% benefit, and peek at the data part of the way through.  If the vaccine turns out to be 90% effective, you’ll be able to stop earlier than planned and declare victory.  You’ll still have recruited more people than if you were betting everything on 90% effectiveness, but you won’t be betting everything on 90% effectiveness.

Stopping early requires strong evidence.  It requires strong evidence because it’s easy to get fooled by randomness in early trends. It requires strong evidence because early stopping means less data on safety. And it  requires strong evidence because people are going to suspect political interference.  One popular guideline for designing early-stopping rules, to give us something concrete to look at, says that you take margin you’d need to be convinced at the end of the trial, and use that margin earlier on.  In our hypothetical example, we had 50 cases difference as the success criterion at the end of the trial, so you’d look for 50 cases difference at earlier timepoints as well.  That’s a much bigger relative difference early on:  if you only had 100 cases, a difference of 50 is 25 vs 75; with only 60 cases, it’s 5 vs 55.

My understanding is that a 90%-effective vaccine would be surprising, but not inconceivable and a 70%-effective vaccine isn’t at all unlikely, so interim analysis and the potential for early stopping do make sense.  Political interference doesn’t help, though: even more than most drugs, a vaccine only counts as effective if lots of people are willing to take it, and a trial that doesn’t engender that willingness is a failure no matter the statistical results.

• A surprising (to me) statistic from the Washington Post: 100 US law enforcement officers have died of COVID contracted at work. That’s more than all other line-of-duty deaths this year put together. And it  doesn’t include 150 more deaths that the Officer Down Memorial Page is still reviewing.
• A particularly malign case of ‘predictive policing’ in Florida.
• You’ve probably heard about the UK’s  plan to impute A-level grades after cancelling exams. A good discussion of why this is a hard problem and how to avoid solving it. 
• “Sooner or later we are going to have to have an inquiry into the role of statisticians in the Covid-19 crisis,” declared the journalist Ross Clark in the Daily Telegraph recently.” David Spiegelhalter would relish the opportunity.
• Alert-level decision in NZ are hard; getting them wrong either way can have very bad consequences. Ars Technica has an excellent report on a similar sort of decision: whether or not to evacuate Houston as Hurricane Laura approached.
• The FDA Commissioner, talking about plasma from recovered COVID patients as a treatment, managed to get both the estimated benefit and the reliability of the evidence badly wrong. Hilda Bastian writes about the implications for informed consent by future patients