Posts written by Thomas Lumley (2645)

• Good piece at Stuff, from The Conversation about COVID statistics
• “The BSA said the decision highlights the importance of data literacy, particularly in a news and current affairs context.” Broadcasting Standards Authority decision, on a complaint about Mike Hosking.
• Two StatsChat-relevant new books: “How to Make the World Add Up” by Tim Harford, and “Calling Bullshit” by Carl Bergstrom and Jevin West
• “Forecast models”…are typically designed with a single goal in mind: to make a specific, quantitative prediction about an event that will be observed in the future….Other infectious-disease “scenario models” are designed to explore multiple “what if” hypothetical futures”.  From the Washington Post
• A story at the Herald, from the Daily Telegraph: low-level exposure to coronavirus through masks  could be giving people immunity. It could be, but there’s little  to no evidence that it actually is.  And we know it isn’t in New Zealand, because there’s almost no coronavirus here to have low-level exposure to, a point that might have been worth mentioning.

Some graphs of the stringency of NZ covid restrictions (from the Oxford Coronavirus Government Response Tracker) because it’s easy to forget what the  rest of the world is like if you don’t  talk to them regularly.

Here is  New Zealand and some countries we get compared to:

That’s not an ideal graph because it’s based on the maximum severity anywhere in the country,  with just a small offset for not being national. Here I’ve tried to work it out as  if Auckland and the Rest of NZ were separate countries. It’s still  not perfect, because other countries also have sub-national variation.

And there is the same comparison for another relevant group of countries

As a bit of light relief from Covid, let’s look at this story at Stuff. It’s mostly about maternity wear (correctly disparaging the maternal-industrial complex), but there’s this

There’s having a face. Yes, you should be ashamed of your natural face. Luckily, there’s all manner of products to correct it, and makeup to cover it up. You’ll need both, to adequately conceal that shameful natural face of yours. It’ll cost the average woman about $400,000 over her lifetime.

If that were true, it would  leave avocado toast in the dust.  Where  does this figure come from?

The story doesn’t say, but there’s a similar figure that’s quite popular, eg at Buzzfeed,

A new survey from Skin Store of more than 3,000 women found that the average woman in the US spends around $300,000 on makeup in her lifetime.

The increase from $300k to $400k looks like it could easily be a currency conversion.  Buzzfeed (of course) doesn’t link to information about  the actual survey, but it is available.

The first thing to notice is

When you take into consideration how much the daily face costs of our New York women, they can spend up to $300,000 per lifetime on skincare products and cosmetics.

So the figure has been misquoted in translation, moving from New York women to US women to NZ women.  It’s probably not true even there — Skin Store don’t give any information about methodology.

We can see how badly off the number must be with a bit of simple arithmetic and a few Google queries. Skin Store says they surveyed women aged 16 to 75. There are about 100 million women aged 18 and over in the US. That’s not quite  the right age range, but it will do as an approximation.  Multiplying by US$300k per lifetime gives US$30 thousand billion.  Dividing by the 60-year ‘make-up lifetime’ gives half a trillion per year.  Total US GDP  is about $20 trillion.  The claim is that more than 2% of total US GDP goes on face-care products. For comparison, the US spends about 0.75 trillion on primary and secondary education, 1.7 trillion on all food.

The Bureau of Labor Statistics, who actually care about getting this sort of thing roughly right, estimate an average annual expenditure of less than US$800 per ‘consumer unit’ on all ‘Personal care products and services’. A ‘consumer unit’ is roughly what normal people would call a household or  family, and $800/year (which includes a lot more than just makeup, and not just women) comes to $48,000 per 60 years.  The Skin Store number looks to be off by around a factor of ten.   Just making things up would be more accurate than that.

There’s a Herald story, from the NYT headlined Covid 19 coronavirus: Vaccine-makers keep safety details quiet, alarming scientists.

The headline  is a bit misleading. The lead says

Researchers say drug companies need to be more open about how vaccine trials are run to reassure people who are skittish about getting a coronavirus vaccine.

which is much more accurate. In fact, the information being sought isn’t mostly about safety but about how effectiveness will be judged

Another front-runner in the vaccine race, Pfizer, made a similarly terse announcement Saturday: The company is proposing to expand its clinical trial to include thousands more participants, but it gave few other details about its plan, including how it would determine the effectiveness of the vaccine in its larger study.

I wrote a bit about early stopping of trials last week. One of the main pieces of information being sought is just how the various trials will handle an unexpectedly good vaccine: how dramatic will the effect of the vaccine have to be to stop the trial early?  Normally, this wouldn’t be public information; there isn’t any compelling reason why it has to  be, but there isn’t any very good reason why not.  This case is different: the political situation has raised a real possibility that  the President of the USA would try to make the FDA authorise a vaccine without convincing evidence that it works. I don’t use words like ‘disaster’ lightly, but this would be a  disaster, both for COVID prevention and for the long-term credibility of vaccination.  We need a vaccine that works, and we need people to know that it works.

Fortunately, the FDA and the pharmaceutical companies are a bit more long-sighted (Pharma, considered as multinational public-stock corporations, is greedy, but they’re not stupid).  If the current administration tries to push a vaccine through without displaying good evidence, there will be resistance from scientists inside and outside the FDA. There are even encouraging signs that the drug companies wouldn’t ask for authorisation without evidence that at least looks moderately convincing from a safe distance.

Even so, it would be good to know the details in advance. We have the FDA Guidance document, which goes some way to nailing the FDA’s colours to the mast. There’s some public information about an NIH working group that seems to have made sensible design recommendations. But in this particular setting it would be helpful to release the details of how the trials will define success — measurements, diagnosis definitions, guidelines for early stopping — so that dozens of scientists with no connection to the trials could say, hand on heart, “that’s not exactly how I would have done it, but it’s 100% a reasonable way to define a successful vaccine”.

Update: in case it isn’t clear, I’m not saying it’s impossible for President Trump to push an ineffective vaccine through the FDA. I’m saying it’s impossible for him to do it without it being obvious to researchers in the US and around the world who he doesn’t control. And that if he doesn’t and the vaccine really works, it’s important for you and everyone else to know that.

Winston Peters, disagreeing with the continued level 2 restrictions (via the Herald)

“Travelling around the South Island has reinforced that people are not observing social distancing in the absence of any registered or real threat of Covid-19 exposure since late April.

“Not because they are against the Government’s Covid-19 response, but because they have applied their own ‘common sense’ test to their risk of exposure to the virus.”

He’s probably right. But that’s the problem.

Our own commonsense understanding of risk is pretty good.  If you’re deciding how fast it’s safe to take a winding road in bad weather, you can do well with commonsense perceptions  (unless you’re drunk or 16, and even  then you’ll probably make it).  If you’re deciding about your personal COVID risk you don’t have as much specific experience to fall back on, but there has been advice around for months and you’ve been watching the daily case counts. So, again, probably yes. The risk is low tomorrow.

But that’s not the question.  We don’t have restrictions because the risk is high. We have them because any COVID transmission will make the risk increase, slowly at first, then faster and faster over time. Since there’s a lag of a week or two in seeing the consequences, things would look the same whether we have a  long  trail dribbling off into nothing, or  a second wave all across the country. I can’t tell the difference. You can’t tell the difference.  Even Siouxsie or Ashley can’t tell the  difference!  In one case, you all in the South Island  have another moderately restricted week; in the other,  you and we go back to lockdown at huge expense and inconvenience, and potentially a bunch of people die or become chronically ill.

Because  the risk is not an immediate and visible and individual one, but a delayed and invisible and community one, commonsense and experience just doesn’t work on its own.   I don’t know whether the decision to extend level 2 was correct or not, because I don’t have all the information and modelling. Mr Peters has seen months of the best scientific and economic advice the country has to offer. He may know. But he’s not arguing based on his special knowledge  but on general commonsense risk perception. That won’t tell you.

One of the COVID vaccine trials sponsored by AstraZeneca has been ‘put on hold’ because of  a ‘suspected serious adverse reaction’.  What does that mean and how bad is it?

Some terminology

• Adverse Event (AE): a bad thing that happens to someone in a trial. These are not necessarily related at all to the trial treatment; something counts as an AE even if you’re getting placebo. They range from minor to serious
• Serious Adverse Event (SAE): the adverse event is hospital bad.  More precisely,  fatal, life-threatening, needing hospitalisation, or leading to significant permanent effect, or  could have gotten that way if it wasn’t stopped by treatment.
• Adverse Reaction (AR): an adverse event that was caused by the trial treatment. Some of the COVID vaccine candidates have quite a  lot of these; fever, for example.
• Serious Adverse Reaction: an SAE that was caused by the trial treatment
• Suspected Serious Adverse Reaction: an SAE that might have been caused by the trial treatment
• Suspected Unexpected Serious Adverse Reaction (SUSAR): an SAE that might have been caused by the trial treatment, and wasn’t disclosed to the participant as a reasonably expected result. (Some treatments do have expected serious adverse reactions — some cancer chemotherapy, for example)

It’s standard in clinical trials that you consider SSARs/SUSARs urgently, and for completely new treatments the consideration should be especially careful.  It might turn out on investigation that the event is unlikely to be caused by the treatment (eg, because of timing, or because you find a more likely cause); it might turn out that it is likely to be caused by treatment;  or everything might just be messily inconclusive.

If you have tens of thousands of people in vaccine trials then adverse events will occur, and some of the adverse events will be serious. That’s inevitable. If you have a treatment that could potentially cause a very wide range of adverse events, then some of the serious adverse events will inevitably look like they could have been caused by the treatment.  And, sometimes, they will have been.  Vaccines aren’t intrinsically safe; they are safe when approved because they get thoroughly investigated first.

The immediate question for the trial sponsor and investigators in this case is whether the adverse event, whatever it is, significantly changes the risk:benefit balance of the vaccine candidate, as explained to the study participants (and to the company’s investors).  So far, no-one knows, but AstraZeneca seem to be handling this appropriately, and there’s no obvious reason to expect that to change.

If you’ve been paying attention to the news you might have read that Forbes ranked NZ second in the world for COVID safety. If you chose more careful news sources, that you might instead have read that Forbes published a story about COVID safety ranking constructed by a company called Deep Knowledge Group.  It’s their second try; they published the first round in June.  And, of course, you might remember stories from the Before Times about New Zealand’s  poor ranking on the Global Health Security Index.

So, which of these indexes are right and which ones are wrong? It turns out that’s the wrong question, just as it is about rankings  of the most liveable city. These rankings aren’t trying to be predictive in a way that makes ‘right’ and ‘wrong’ objectively assessable.  What happens is that a group of people gathers together a whole lot of measurements.  These measurements range from ‘good’ to ‘bad’ on some scale. Points get awarded for how close you are to ‘good’ on each measurement and then added up. The experts then look at the rankings they got out, and probably adjust the points a bit to  make them look more plausible — there’s nothing wrong with this; the experts’ judgment is where there’s potentially value added by having this index.

There is a technical problem with reducing things to a sum of numbers in this way — and I say that as someone who likes numbers. We might all agree that, say, willingness to follow mask recommendations is positive, and that having a policy of mask recommendations when appropriate is positive. But these don’t add up: you need both of them.   Being more willing to follow government advice to wear masks does no good if the government is not giving that advice; having the government give the advice does no good if it isn’t followed.  You can’t represent A AND B as a weighted sum.  Weighted sums are a very limited subset of ways of combining ordered scales.

It’s clear how much opinion goes into the index construction when you see that the Global Health Security Index and the new COVID index share only two countries in their top ten: Australia and South Korea.  The United States and the UK top the GHS Index; Germany and NZ top the Deep one.  The real question is whether the indexes are useful, and they probably are — or, more precisely, whether the reports surrounding the indexes are useful, where the experts talk about the ways countries differ and their strengths and weaknesses.

If you wanted to think of the indexes as predictive, you’d want to look at the June edition and see how it’s fared.  Most of the top ten have done ok, but Israel, at number 3,  has had a bad few months, and now has 27,000 active cases in a country with less than twice the population of New Zealand.  They have been demoted to number 11: four places ahead of Taiwan, 21 ahead of Vietnam, and 83 ahead of Mongolia.

The index and its surrounding report contain useful information; for example they identify (admittedly with 2020 hindsight) the importance of political will to act and social acceptance of restrictions.  The ranking of NZ as the second-safest country is only meaningful if you don’t interpret ‘safest’ in some crudely reductive way as referring to the likely number of cases or hospitalisations or deaths.

There are about forty COVID vaccine candidates now, with several heading to Phase 3 trials.   The  CDC has suggested there could be a vaccine before the US presidential election. How could that happen? There’s one obvious answer, sadly, but how else could it happen?

The FDA has said it would expect that a COVID-19 vaccine would prevent disease or decrease its severity in at least 50% of people who are vaccinated.  In the current circumstances, if your vaccine was that effective it would be false economy to run a trial that was too small — you’d want to be very confident of a positive result if your vaccine met the mark.

You might design a trial that ran until you had seen, say, 150 infections or symptomatic cases or whatever. If you ended up with a split of 50 vs 100, you’d estimate a 50% reduction with the vaccine, and the uncertainty interval (‘margin of error’) in would stretch to a 30% reduction, which correspond to the FDA’s guidance.   Only 150 cases perhaps doesn’t sound like much, but remember that at a very high infection rate of 1% you’d need 10,000 people in the control group and the same number in the treatment group. That’s a big trial.

It’s possible, though, that your vaccine is much better than just a 50% reduction in cases. If it actually prevents 90% of cases, you could have run a much smaller trial, at lower cost, finished earlier, and saved more lives.  On the other hand, if you ran the smaller trial and the effectiveness is only 50%, you are in big trouble — you won’t get approved, but you’ll find it harder to run another trial.

So, what we do is to set up a big trial that can detect a 50% benefit, and peek at the data part of the way through.  If the vaccine turns out to be 90% effective, you’ll be able to stop earlier than planned and declare victory.  You’ll still have recruited more people than if you were betting everything on 90% effectiveness, but you won’t be betting everything on 90% effectiveness.

Stopping early requires strong evidence.  It requires strong evidence because it’s easy to get fooled by randomness in early trends. It requires strong evidence because early stopping means less data on safety. And it  requires strong evidence because people are going to suspect political interference.  One popular guideline for designing early-stopping rules, to give us something concrete to look at, says that you take margin you’d need to be convinced at the end of the trial, and use that margin earlier on.  In our hypothetical example, we had 50 cases difference as the success criterion at the end of the trial, so you’d look for 50 cases difference at earlier timepoints as well.  That’s a much bigger relative difference early on:  if you only had 100 cases, a difference of 50 is 25 vs 75; with only 60 cases, it’s 5 vs 55.

My understanding is that a 90%-effective vaccine would be surprising, but not inconceivable and a 70%-effective vaccine isn’t at all unlikely, so interim analysis and the potential for early stopping do make sense.  Political interference doesn’t help, though: even more than most drugs, a vaccine only counts as effective if lots of people are willing to take it, and a trial that doesn’t engender that willingness is a failure no matter the statistical results.

• A surprising (to me) statistic from the Washington Post: 100 US law enforcement officers have died of COVID contracted at work. That’s more than all other line-of-duty deaths this year put together. And it  doesn’t include 150 more deaths that the Officer Down Memorial Page is still reviewing.
• A particularly malign case of ‘predictive policing’ in Florida.
• You’ve probably heard about the UK’s  plan to impute A-level grades after cancelling exams. A good discussion of why this is a hard problem and how to avoid solving it. 
• “Sooner or later we are going to have to have an inquiry into the role of statisticians in the Covid-19 crisis,” declared the journalist Ross Clark in the Daily Telegraph recently.” David Spiegelhalter would relish the opportunity.
• Alert-level decision in NZ are hard; getting them wrong either way can have very bad consequences. Ars Technica has an excellent report on a similar sort of decision: whether or not to evacuate Houston as Hurricane Laura approached.
• The FDA Commissioner, talking about plasma from recovered COVID patients as a treatment, managed to get both the estimated benefit and the reliability of the evidence badly wrong. Hilda Bastian writes about the implications for informed consent by future patients

There’s an interesting poll in the NZ$ Herald asking whether the four-day extension to the Auckland level 3 ‘lockdown-lite’ was appropriate or not. Here’s a graph of the regional results (excluding the worryingly-small fration of “don’t knows”). The purples are ‘yes, should have been extended’, with the light purple being ‘the four day extension was appropriate’ and the dark being ‘longer would have been better’. The oranges are ‘no, should not’, with the light orange being ‘should not have been extended’ and the dark being ‘should not have happened at all’

As you can see, there’s pretty strong consensus across the country.  You’d expect people outside Auckland, who get the benefits but with less of the cost, to want tighter restrictions, and the patterns seem to fit that.

The Herald also explores differences by age, gender, and income. It’s hard to say anything too strong about many of the differences, because we’re only quoted a margin of error for the whole survey, not for any of the subgroups.  In some cases I can work it out: if Auckland and Canterbury were represented in the sample in proportion to their actual population sizes, the difference between the purple bars would be right about at the margin of error.  In general, though, the overall margin of error is pretty useless for most of what makes the story interesting.

Oh, and also.

Auckland is strongly divided over whether extending the lockdown was an appropriate response to the resurgence of Covid-19, a new poll shows.

But the exclusive new poll shows rest of the country was far more accepting of the Super City being kept in alert level 3 for almost three weeks – with many wanting it extended even longer.

Emphasis added.

No. That’s really not what the poll says