Posts written by Thomas Lumley (2645)

At the moment, NZ policy is to test everyone with suitable symptoms consistent with COVID-19, and potential contacts of cases, but not to go around randomly bothering healthy people for community surveillance.  Looking at some numbers explains why that’s a good strategy, and also gives a way to think about what it takes for other surveillance strategies to be useful.

The first question is how well you do just by testing symptomatic people, when we know some people never develop symptoms, and other people only develop symptoms after passing on the virus.  Professor Nick Wilson and various co-workers^* studied this problem back in May.  They did a lot of computer simulations of what would happen if you introduced one COVID case to ‘an island nation’ where the coronavirus had been eliminated but there was still widespread testing.  Under the assumption that about 40% of cases ended up getting tested, they found that an outbreak had a fifty-fifty chance of being detected when there were only six active cases, but that a reasonable worst case was 50-100 active cases at the time of detection.

You can disagree with the particular assumptions being made (they did this way back in May, the coronavirus equivalent of the Sony Walkman era) but it’s a reasonable ballpark guide.   The idea is that you maybe don’t routinely test absolutely everyone with a cold, but you do test everyone with some (new or worsening) respiratory symptom plus shortness of breath, or fever, or loss of sense of smell, or various other combinations.  We’ve gotten lucky: flu-like illnesses are much less common so far this year than in a usual year, so right now we don’t need to test as many people as they modelled.

So, taking the reasonable worst case, suppose at some point there are 50-100 people out there in NZ with coronavirus and we’ve been unlucky enough that none of them got tested (or a few got tested and the tests were false negatives).  How many random people would we need to test to pick up this outbreak?

Fifty people in NZ is one person in 100,000, so we’d need to test about 100,000 people to have a chance of finding a case.  A simple statistical rule of thumb says that testing about 300,000 people would make us pretty sure to find a case.  Outbreaks grow fast;  if there’s an outbreak with 50 people this week, it had maybe  15 people last week. To get a worthwhile improvement in detecting even the worst outbreaks we’d need to test 100k-300k healthy people each week. That isn’t happening.  Random community testing could be useful if we knew where to look. If we had one suspected case in a town of 10,000 people it might be worth just testing as many people as we could, to try to  get ahead of the contact-tracing process. But if you don’t know where to look there isn’t much point in looking.

Sewage testing is another promising possibility for picking up outbreaks, but these numbers show that it’s not going to be easy.  The testing has to be reliable enough that we’d be prepared to take some fairly major and expensive actions based on finding the virus, but sensitive enough to pick up just 50 or so cases.  It currently isn’t clear whether or not that’s possible, but ESR have the expertise (and some funding) needed to work on the question.  Reliable wastewater testing would be very helpful in the situation we’re now in, where there’s a  suggestion of transmission in NZ but not good evidence — but unreliable wastewater testing would just make things worse.

The take-home message is that we’re probably going to find the next outbreak by testing someone with symptoms. That person might very well have no known contact with international travellers.  If you might be that person, you should call Healthline to ask about getting tested.

^* Statisticians will recognise Matt Parry; any Kiwi who hasn’t been hiding in a cave on Mars with their fingers in their ears should recognise Ayesha Verrall and Michael Baker.

We aren’t getting international tourism at the moment, which is obviously a problem for those working in the international tourism industry¹, and to some extent a problem for everyone because of the hit to the economy.

I saw some speculation on Twitter today about how big international tourism actually is, and about the extent to which NZ tourism expenditure staying in NZ would offset the losses.   Now, there will obviously be gaps, where foreign and domestic tourists don’t do the same things (eg, domestic tourists don’t buy long-distance plane tickets from Air New Zealand), but what about the totals?

Overall, tourism (as defined in the tourism satellite account) brought in $17 billion in the year ending June 2019.  Nearly $4 billion of it was actually international education lasting less than a year, leaving a bit over $13 billion in ‘real’ tourism.  That’s just behind dairy, but roughly equal to meat and wood products together.  The short-term international-education component of ‘tourism’ was a bit a head of fruit exports.

Domestic expenditures on international tourism aren’t completely captured, but the Household Expenditure Survey estimates that all NZ households together spent $2 billion on “overseas accomodation prepaid in NZ” and $4.5 billion on “international air transport” in the year to June 2019.  That’s going to miss food bought overseas and admission tickets to cultural experiences, and some overseas accomodation, but it still looks as though redirecting NZ tourism locally would leave a big hole.  The Household Expenditure Survey does miss out on business travel that isn’t a household expenditure, but it seems more of a stretch that business travel spending will just be redirected to NZ.

¹ I was surprised to find this, in one sense at least, includes me, since international students here for less than 12 months are counted in the ‘tourism satellite account’.

The recent trends in opinion-poll support for the National Party got a lot of attention. That’s because real opinion polls, like those done by Colmar Brunton and Reid Research (and the internal party polling that they tell us about when they think it will help them) are genuine attempts to estimate popular opinion.  You can argue about how good they are — but you can argue about how good they are, there are factual grounds for discussion.

NewsHub ran a bogus online clicky poll with the question Who would you prefer as Prime Minister – Judith Collins or Jacinda Ardern?  Of the people who clicked on the poll, 53% preferred Ms Collins, and 47% preferred Ms Ardern.  Let’s compare that to the two real polls. The 1News/Colmar Brunton poll had

• Jacinda Ardern: 54% 
• Judith Collins: 20%

The 3/Reid poll had

• Jacinda Ardern: 62% 
• Judith Collins: 15% 

Why are these so different from the Newshub clicky poll? The first point is that there’s no reason for them to be similar. Two of them are estimates of popular opinion; the other one is a video game.

On top of that, the question is different.  The real polls are asking who (out of basically anyone) is your preferred PM.  The bogus poll forces the choice down to Ardern vs Collins.  If you supported Simon Bridges or Todd Muller — or Metiria Turei  or Winston Peters — the real polls let you say so, and the bogus poll doesn’t.

Research Association NZ, who are the professional association for opinion researchers in NZ, have a code of practice for political polling (PDF). It’s only binding on their members, but it does have best practice advice for the media, such as using the term “poll” only for serious attempts to estimate public opinion, not for bogus clicky website things.

• The Algorithm Charter has been released. Stories from NewsHub, newsroom, The Guardian, The Register, ZDnet
• Covid-19 in Victoria: bad, and according to modelling by Peter Ellis, still getting worse.  If you’re in NZ, make sure you have a mask and hand sanitiser available and at least have the contact apps on your phone, in case we get another outbreak.
• RadioNZ’s podcast “The Detail” has an episode on polling.  I’m reliably informed that I’m on it.
• In 2020, we’re amid that critical juncture for ’90s music—we can finally start asking today’s teens, “What music do you recognize from the ’90s?”. From pudding.cool

A company called Genderify has what they say is “an AI-powered tool for identifying the gender of your customers”. This is an example of something that is not worth doing (asking is easy and reliable; people will be upset when you get it wrong), but also very difficult.

After seeing some examples on Twitter, I decided to try it on some senior members of the Stats department (whose gender identity I’m reasonably confident of)

“Thomas Lumley” is 63.90% likely to be male and 36.10% likely to be female, and you have to like the four digit precision. But “Dr Thomas Lumley” is 89.40% likely to be male, and “prof thomas lumley” gets up to 94.60%!

“Ilze Ziedins” is 85.20% likely to be male, which will surprise her. “Dr Ilze Ziedins” gets to 96.00%

“James Curran” is 99.60% likely to be male, adding “Dr” or “Prof” gets him up to 99.90%

“Rachel Fewster” is at 72.00% likely to be male, adding her professorial title puts that up to 95.40%

“Renate Meyer” is at 62.30%, her doctorate moves that up to 88.20%, and her promotion to professor makes it 94.00%

Note that none of these are classically gender-neutral or gender-ambiguous names: no Hadley or Hilary or Cameron.  The overall level of accuracy is pretty terrible to start with — but the response to adding qualifications is bizarre.  If that wasn’t in the basic pre-release testing, then what was?

Even better (worse):  it’s not just that adding “Dr” or “Prof” make it think you’re more likely mean a man, adding “Dame” also does.

Update: on Twitter, (Dr, Prof) Casey Fiesler raised the possibility that Genderify are just trolling, which I must say is looking quite plausible.

It’s getting to be that time of the triennium again, so some highlights from past StatsChat posts on electoral polling

• Overview of polling and how it works
• What is the ‘margin of error’ for minor parties: one, two
• Why you can’t predict Epsom from polls
• Why polling is important: your friends aren’t representative
• Survey weighting: why polls don’t massively underestimate the Greens vote.
• The difference between 1 in 20 exceedance of the margin of error and a ‘rogue poll‘

I wrote about ordinary sampling variation and ‘rogue polls’ for The Spinoff

In clinical trials we make two important distinctions between measurements. There are ‘hard’ and ‘soft’ outcomes — ‘hard’ ones are objectively and reproducibly measurable, ‘soft’ ones have some subjectivity and observer bias.  There are also ‘surrogate’ and ‘real’ or ‘patient-centered’ outcomes. ‘Real’ outcomes are what we care about; ‘surrogate’ outcomes are things we measure because we can measure them well and we expect them to correlate with real outcomes. Hard and soft outcomes are valuable; real and surrogate outcomes are valuable; you don’t want to confuse them.

There’s a story on NewsHub headlined Bisexual men are real, study finds. One of the researchers had previously doubted this, but has now been convinced, and has paper in PNAS about an analysis combining data from many previous studies.  These studies involve wiring someone’s penis up to detect arousal and then showing him erotic images.  The claim is that these measurements are objective

If men who self-report Kinsey scores in the bisexual range indeed have relatively bisexual arousal patterns, then both Minimum Arousal and the Bisexual Arousal Composite should show an inverted U-shaped distribution across the Kinsey range (i.e., men who self-identify as 0 [exclusively heterosexual] and 6 [exclusively homosexual] should have the lowest scores for these variables; men in intermediate groups should have greater values, with the peak resting at a Kinsey score of 3); the Absolute Arousal Difference should show a U-shaped distribution (i.e., exclusively heterosexual and exclusively homosexual men should have lower values than bisexual-identified men).

The reason for emphasising these measurements is that they doesn’t completely trust self-report (while agreeing it is valuable)

However, because the scale relied on self-reports, results could not provide definitive evidence for bisexual orientation. For example, surveys have shown that a large proportion of men who identify as gay or homosexual had gone through a previous and transient phase of bisexual identification 

I don’t think anyone (whatever their opinion on bisexuality) would deny that men who lie about sex are real. The problem is treating the physical arousal measurements as basically definitive of bisexuality.  In the clinical trials terminology, the arousal measurement is a relatively hard outcome, but it is a surrogate outcome.

With modern data science (and sufficiently dodgy ethics) there would be other surrogate outcomes that someone has probably explored.  Are there a significant number of men on Tinder who swipe right for both male and female profiles?  Are there many PornHub accounts of men who watch both straight and gay porn? Are there men who have shared a one-bedroom home with both men and women over time?  All of these are clearly reductive: they would give you one-dimensional information about bisexuality, but they are measuring different things and there’s no reason to expect they would agree on how common it is.  The same is true for physiological arousal.  Measuring it can be valuable; the demographics of physiological arousal can be a valid area of study; but it can’t answer the yes/no question.

Some men claim to be attracted to both men and women, and behave as if their claims are true. It turns out, according to this paper, that for some of these men the physiological measurements of arousal show the relationships that you’d expect.  If there weren’t any men whose physiological measurements of arousal show those relationships, that would be an interesting fact, but the real question would be why the measurements don’t fit with the phenomenon of bisexuality.  If you think of this paper as just trying to answer a question about physiological arousal then, ok, that’s the question it tries to answer. And in fact one of the researchers is quoted further down in the NewsHub story saying

“It has always been clear that bisexual men exist in terms of self-identity and behaviour, but many, including myself, were sceptical about their ability to be sexually aroused to both men and women.” 

Contrast that, though, with the paper’s “Significance” section, which starts out

“There has long been skepticism among both scientists and laypersons that male bisexual orientation exists.”

Or with the second sentence of the press release:

“The existence of male bisexuality is contested, with skeptics claiming that men who self-identify as bisexual are actually either homosexual or heterosexual.”.

Or with the title of the research paper itself

Robust evidence for bisexual orientation among men

The stretching of the study findings to the headline “Bisexual men are real, study finds” can’t just be blamed on the media.

When we talk about whether Alexander the Great or Shakespeare was bisexual, there are difficulties in even agreeing on the concept over centuries or millennia of social distance.  But I think most people would agree there’s more to the question than what would have happened if you wired them up to a machine and showed them porn.

• Interpreting coronavirus numbers, from Pro Publica
• Pretty good Herald story with terrible headline: Covid-19 coronavirus: Superspreader – woman infects 71 people in 60 seconds in elevator: CDC study. The woman in the elevator wasn’t by any definition a ‘superspreader’. She infected one person.  They don’t link to the CDC study.
• Nice explanation of PCR testing for the coronavirus, from U Otago
• Stats NZ are asking for input on their proposed standards for asking about gender and sex.
• Wikipedia is a good source of information on COViD-19. Mike Dickison talks to The Spinoff about why this is.

Non-representative sampling!

The results are in! #MostLoved #RollTide pic.twitter.com/IacOVUhFHL

— Alabama Athletics (@UA_Athletics) July 14, 2020

Time magazine has a new story Popular Heartburn Drugs Linked to Heightened COVID-19 Risk, and it’s on Reuters, so it’s going to spread.  Here’s the preprint, and here’s the press release, which comes from the American College of Gastroenterology, the major professional association in that field and the journal publisher.

The drugs in question are proton-pump inhibitors (PPIs). These inhibit the cellular pumps that push hydrogen ions into the stomach fluid, making it acid. If your stomach acid isn’t acid enough, it’s plausible that the coronavirus could survive and get into your gut, where it will find cells with the receptors it needs to attack.  So the theory is not at all unreasonable.  Kiwis will remember that Michelle Dickinson and Siouxsie Wiles both emphasised the potential risk of coronavirus getting in through your mouth.

The press release says “We have now tested the hypothesis in a rigorous study of more than 50,000 Americans and found it to bear out, albeit in an observational study.”  As you’ve probably guessed, I’m not buying that.

In the research paper, the study is described as

we used data from a population-based, online, self-administered survey of Americans collected from May 3 to June 24, 2020. We collaborated with an online survey research firm (Cint) that recruited a nationwide, representative sample based on U.S. Census data on age, sex and region

That could work, provided it really was a representative sample, and provided you could get a good handle on the other reasons why taking a medication long-term might be correlated with getting a positive COVID test.

The sample of 54,000 people included 3,386 (6.4%) who reported having had a positive COVID test.  As of June 24, only 2.35 million people in the US had tested positive for COVID. There are about 250 million adults in the US, so even if all the positive tests had been in adults, that’s less than 1%.   The sample has over six times the national average for positive COVID tests. Regional bias wouldn’t be enough to explain this: even in New York City, only about 2.5% of the population has tested positive.

Even with this very high rate of COVID, the sample is missing a lot of cases. 95% of people with positive tests reported symptoms. That’s not surprising, as symptoms are how you get tested, but it does mean the ‘control’ group will contain at least another few thousand cases who didn’t get tested, or got tested at the wrong time.

If you just look at the PPI data, 75% of people with a COVID diagnosis were regularly taking PPIs, as were 30% of the other participants.  In a population-based health survey that makes serious efforts to be representative, NHANES, only 8.7% of participants reported taking PPIs.

So that’s the overall representativeness. The other thing to worry about is that regularly taking a medication and getting access to a COVID test may well be correlated, so you’d worry about whether the COVID cases were different in other ways — remember, we already know sampling was far from representative.

A few differences jump out. The COVID cases were more than 8 times as likely to have a household annual income over $200,000: nearly two-thirds of them did. That’s despite them being less likely to have a college degree.  More than two-thirds of the cases reported Latinx/Hispanic ethnicity, but only 3.5% were non-Hispanic Black. Only 10% of cases were from the Northeast of the US, where the epidemic has been worst until recently;  nearly 70% were from the South.  The cases were much less likely to report a diagnosis of gastroesophageal reflux disease, a primary reason for taking PPIs.

The researchers did make some efforts to adjust for the non-representative sampling.  The relative risk of COVID in people taking PPIs daily or less often went down from nearly 8 before adjustment to 2.15. For people taking PPIs twice a day, the relative risk went down from 5.7 to 3.7.  However, the researchers didn’t use a lot of the variables in this adjustment, and they didn’t try to reweight the data to known national proportions for age, race/ethnicity, gender, and region, a fairly standard technique in national surveys (eg, election polls, market research).

Given the clearly non-representative sample, I don’t think the evidence could be convincing without a lot more exploration of the biases (and quite likely not even then).  As a drug class, PPIs have form for this: there have been other conditions correlated with PPI use in initial reports, where the correlations go away with better-quality data.

I’m not saying the study shouldn’t have been done, though I think it should have been analysed better. But the journals shouldn’t have pushed it out into the media with an urgent pre-publication press release, especially when even the authors won’t publicly claim it’s good enough evidence to change treatment.  The true take-home message of this study, apparently, is that it gives doctors an opportunity to stress the importance of hand-washing.

If I were more cynical than I am, I would have pointed out much earlier in this post that one of the authors of the study  is co-Editor-in-Chief of the journal, and is quoted in the press release with that title.