Stats Chat

In 2012, the Telegraph and other UK papers were off by five orders of magnitude when they said there were only 100 adult cod in the North Sea. The Washington Post beats that easily.

Quantum computers are straight out of science fiction. Take the “traveling salesman problem,” where a salesperson has to visit a specific set of cities, each only once, and return to the first city by the most efficient route possible. As the number of cities increases, the problem becomes exponentially complex. It would take a laptop computer 1,000 years to compute the most efficient route between 22 cities, for example. A quantum computer could do this within minutes, possibly seconds.

As mathematician Bill Cook pointed out on Twitter, his iMac can solve a 22-city problem in 0.005 seconds, roughly six trillion times faster than the Washington Post claims.  It looks as though the writer has assumed there is no more efficient algorithm than just trying all the possible routes, one at a time.  At one billion attempts per second, that would take on the order of a thousand years.  In fact, there are more efficient algorithms, it’s just that these algorithms also get very slow as the number of points increases. Prof Cook estimates in the twitter thread that 1000 CPU-years might allow a 100,000-point problem to be solved: six trillion times more computing gets you only a 5000-fold increase in the number of points.

For the cryptographic problems that are the actual point of the story, fast quantum algorithms are known. If large enough quantum computers can be built, these security protections are toast. And as the story says, there’s research going on now to find replacements if/when they’re needed (though what the story says about those algorithms is almost completely unhelpful).  But the travelling salesman problem is not one of the ones for which fast quantum algorithms are known. On the contrary, there are reasons to suspect fast quantum algorithms don’t even exist for “NP-complete” problems such as the travelling salesman.

Update: Scott Aaronson, who knows from quantum, is Not Impressed.

Team Ratings for Round 1

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Predictions for Round 1

Here are the predictions for Round 1. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Ipsos MORI do a nice set of surveys about public misperceptions: ask a sample of people for their estimate of a number and compare it to the actual value.

The newest set includes a question about the proportion of the prison population than are immigrants. Here’s (a redrawing of) their graph, with NZ in all black.

People think more than a quarter of NZ prisoners are immigrants; it’s actually less than 2%. I actually prefer this as a ratio

The ratio would be better on a logarithmic scale, but I don’t feel like doing that today since it doesn’t affect the main point of this pointpost.

A couple of years ago, though, the question was about what proportion of the overall population were immigrants. That time people also overestimated a lot.  We can ask how much of the overestimation for the prison question can be explained by people just thinking there are more immigrants than there really are.

Here’s the ratio of the estimated proportion of immigrants among the prison population and the total population

The bar for New Zealand is to the left; New Zealand recognises that immigrants are less likely to be in prison than people born here. Well, the surveys taken two years apart are consistent with us recognising that, at least.

That’s just a ratio of two estimates. We can also compare to the reality. If we divide this ratio by the true ratio we find out how much more likely people think an individual immigrant is to end up in prison compared to how likely they really are.

It seems strange that NZ is suddenly at the top. What’s going on?

New Zealand has a lot of immigrants, and we only overestimate the actual number by about a half (we said 37%; it was 25% in 2017). But we overestimate the proportion among prisoners by a lot. That is, we get this year’s survey question badly wrong, but without even the excuse of being seriously deluded about how many immigrants there are.

•  Aaron Schiff’s maps of “Small Towns of Aotearoa”
• “The Shame of Being a Statistic”  by Diana Hennessy
• Color palettes: pigment.shapefactory.co (via @fogonwater)
• “From Spreadsheets to Headlines: A Day in the Life of Data Journalist Ryan Struyk“
• Bad data: What happens when a huge number of people share a single grocery store loyalty card? (not much, actually)
• Felix Salmon approves of changes in how Oxfam reports its wealth inequality data.
• “nobody thought to explain to it the long history of Black people being dehumanized by being compared to apes. That context is what made this error so serious and harmful, while misidentifying someone’s toddler as a seal would just be funny”. On AI and its limitations — not a new topic, but an usually good examination.
• In which everywhere is above average: data analysis of tourist expenditures by Peter Ellis.
• Matt Nippert and Keith Ng have a session on data journalism at the New Zealand Festival, if you’re in Wellington on March 9
• Key Concepts for Informed Health Choices: the website and the research paper. ” The concepts in this list are universally relevant. However, our initial focus has been on concepts that are important and possible to teach or communicate to children and poorly educated adults living in low-income countries. “

I’ve seen this map several times on Twitter. It’s originally from a British company, ThirtyFifty. It gives a really nice idea of where the world’s wine regions are, and why, in terms of temperature.

There’s one problem, which is probably more obvious to New Zealanders than northern-hemisphere people. The latitude lines are wrong.

The 50N line and the equator are right, but the 30N line is a little too far north; the 30S line is a little too far north; and the 50S line is way too far north — it’s at about 45S, measured by the southern tip of Tasmania and the middle of NZ’s South Island.

The New York Times and Stuff both have recent stories about fake Twitter followers. There’s an important difference. The Times focuses on a particular company that they claim sells fake followers; Stuff talks about two apps that claim to be able to detect fakes by looking at their Twitter accounts.

The difference matters. If you bought fake followers from a company such as the one the Times describes, then you (or a ‘rogue employee’) knew about it with pretty much 100% accuracy.  If you’re relying on algorithmic identification, you’d need some idea of the accuracy for it to be any use — and an algorithm that performs fairly well on average for celebrity accounts could still be wrong quite often for ordinary accounts. If you know that 80% of accounts with a given set of properties are fake, and someone has 100,000 followers with those properties, it might well be reasonable to conclude they have 80,000 fake followers.  It’s a lot less safe to conclude that a particular follower, Eve Rybody, say, is a fake.

Stuff says

As StatsChat readers will know, there are lots of candidate tests for Alzheimer’s Disease, all of which are much better than just flipping a coin. These tests may be useful in selecting people for clinical trials, and if we ever get effective disease-modifying treatments, for deciding who to treat. At the moment, though, these tests aren’t much use.

BBC News has another one

Scientists in Japan and Australia have developed a blood test that can detect the build-up of toxic proteins linked to Alzheimer’s disease.

The work, published in the journal Nature, is an important step towards a blood test for dementia.

The test was 90% accurate when trialled on healthy people, those with memory loss and Alzheimer’s patients.

If you’re not paying careful attention, 90% accuracy sounds pretty good. But that’s the accuracy in a group of people where about half of them have Alzheimer’s.  In the population, where most people are ok, the false positive rate will still be scarily high.

Also, in contrast to some tests I’ve written about, the main focus of this paper is differentiating Alzheimer’s Disease from other sorts of dementia

The research paper says

The plasma composite biomarker showed 96.7% sensitivity, 81.0% specificity, and 90.2% accuracy in the overall data (n = 51) when predicting individual Aβ status (Aβ⁺ or Aβ⁻) using the common cut-off value (0.376) (Extended Data Fig. 8e–g). The results suggest that the plasma biomarker could be helpful for the differential diagnosis of AD and aid in determining therapeutic strategies, by providing additional information on the brain Aβ deposition status of individuals.

That’s going to be useful when we get treatments, since the treatments for Alzheimer’s probably won’t work on unrelated kinds of dementia, but it doesn’t really fit with the framing in the story

Alzheimer’s disease starts years before patients have any symptoms of memory loss.

The key to treating the dementia will be getting in early before the permanent loss of brain cells.

• “Are bigoted, irrational robots hurting humans?” (Herald, from the Daily Mail).
• From 538: ” If I had a magic wand, I’d develop an algorithm that: a) draws the shortest possible line(s) necessary to split a state into equally populous districts and b) requires that only as many pre-existing jurisdictions be split as necessary to achieve equally populous districts.” That’s the easy part. Getting a fair voting system isn’t primarily difficult because math is hard, but because power corrupts.
• Sharon Begley at STAT on the hype around exercise and dementia “But we thought it was a reasonable thing to say, especially since [exercise] doesn’t have a lot of risk. To be honest, we’re looking for something positive to tell people.”
• Peter Ellis writes on how to recruit data scientists in the public sector
• There’s a famous medical paper reviewing all the randomised trials of parachute use when jumping from a plane (the point being, of course, that there aren’t any, and shouldn’t be). There’s a new medical paper looking at people using this analogy. Of 35 interventions where the parachute analogy was used explicitly to argue that randomised trials were impossible and there was no room for doubt about effectiveness , 22 had been examined in randomised trials. Six had been shown effective.

From Stuff The Herald:  ‘Potentially hazardous’ asteroid heading towards Earth at 108,800km/h, and in the story

It is the largest space rock to brush past our planet this year and previous research has found a rock of this size could plunge Earth into a mini-ice age if it hit.

The impact would cause average temperatures around the world to fall by as much as 8°C, according to a 2016 study on the effects of a collision with a 0.6-mile-wide (1km) asteroid.

“These would not be pleasant times,” Charles Bardeen, of the National Center for Atmospheric Research, said during a presentation at the American Geophysical Union (AGU).

In the “worst case scenario”, soot would remain in the atmosphere for around 10 years, while dust take six years to settle back on Earth.

Fortunately Nasa does not think this asteroid will collide with Earth.

NASA would put that more strongly. From NASA/JPL Asteroid Watch: Asteroid 2002 AJ129 to Fly Safely Past Earth February 4,

At the time of closest approach, the asteroid will be no closer than 10 times the distance between Earth and the Moon (about 2.6 million miles, or 4.2 million kilometers).

“We have been tracking this asteroid for over 14 years and know its orbit very accurately,” said Paul Chodas, manager of NASA’s Center for Near-Earth Object Studies at the Jet Propulsion Laboratory, Pasadena, California. “Our calculations indicate that asteroid 2002 AJ129 has no chance – zero – of colliding with Earth on Feb. 4 or any time over the next 100 years.”

The @asteroidwatch twitterwallah is off because of the government shutdown, but was relatively patiently answering Twitter questions in both English and Spanish until Friday.

Part of the problem is that NASA uses “potentially hazardous” for any asteroid approaching within 1/20th of the distance to the sun, because these are the ones they want to track to be sure that they aren’t actually hazardous.  It’s an unfortunate term because it sounds dangerous when it isn’t.

• On herd immunity, from Quanta magazine “The necessary level of immunity in the population isn’t the same for every disease. For measles, a very high level of immunity needs to be maintained to prevent its transmission because the measles virus is possibly the most contagious known organism.”
• Pornhub, as is its habit, has an interesting graph on what the Hawai’i missile false alarm did to its workload (not very NSFW, but Pornhub)
• From NPR For Now, Sequencing Cancer Tumors Holds More Promise Than Proof.  Even though the story is about how DNA sequencing of tumors hasn’t been a very successful strategy so far, they lead with a ‘miracle cure’ story, because that’s how the news works.
• “Bad Graphics: manipulation or laziness” from David Spiegelhalter, talking about an example from the Daily Mail.
• Also from David Spiegelhalter: there’s a recent research paper on correlations between alcohol consumption and dementia. The researchers said ““Consuming more than one UK standard unit of alcohol per day is detrimental to cognitive performance and is more pronounced in older populations”, which was widely reported in the media. This is their graph of their analysis. The minimum of the graph is at 16g/day: two UK units or 1.6 NZ units.
  
  If we could trust this model, it would actually say consuming less than one unit per day is detrimental.
• Via Elaine Smid, the  title and abstract of a research paper