Stats Chat

• “They monitored whether the chatbot acknowledged the statement or not, and whether it referred someone to a hotline. Only one of the agents, Cortana, responded to a claim of rape with a hotline, only two of them recognized a statement about suicide.”  From freedom-to-tinker

• “China is building the world’s most powerful facial recognition system with the power to identify any one of its 1.3 billion citizens within three seconds.” South China Morning Post

• “Pornhub announced that it is using machine learning and facial recognition“. They say it’s to improve search, according to Vice, which is more or less why people are worried.

• From the Herald: “ACC has paid out on 660 claims where (pedestrian) cellphone distraction has been noted as the injury’s cause.”  You, of course, are asking what that comes to as a percentage of car crash costs. Outsourced to @aw_nz on Twitter

• “Google News Lab is inviting artists and designers to dig into its massive trove of data about what the world Googles.” (from fastcodesign). Five of the the results are shortlisted for the Information is Beautiful awards: Rhythm of Food, I’m not feeling well, How to fix a toilet, Beautiful in English. (via Alberto Cairo)

Team Ratings for the Mitre 10 Cup Semi-Finals

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 70 matches played, 48 of which were correctly predicted, a success rate of 68.6%.
Here are the predictions for last week’s games.

Predictions for the Mitre 10 Cup Semi-Finals

Here are the predictions for the Mitre 10 Cup Semi-Finals. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for the Currie Cup Final

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Note that Cheetahs2 is the Cheetahs team in weeks when the first team is playing in the Pro14.

Performance So Far

So far there have been 42 matches played, 28 of which were correctly predicted, a success rate of 66.7%.
Here are the predictions for last week’s games.

Predictions for the Currie Cup Final

Here are the predictions for the Currie Cup Final. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Each week, we would like to invite readers of Stats Chat to submit nominations for our Stat of the Week competition and be in with the chance to win an iTunes voucher.

Here’s how it works:

• Anyone may add a comment on this post to nominate their Stat of the Week candidate before midday Friday October 20 2017.
• Statistics can be bad, exemplary or fascinating.
• The statistic must be in the NZ media during the period of October 14 – 20 2017 inclusive.
• Quote the statistic, when and where it was published and tell us why it should be our Stat of the Week.

Next Monday at midday we’ll announce the winner of this week’s Stat of the Week competition, and start a new one.

(more…)

Sam Warburton (the economist, not the rugby player) has been writing about the recent increase in road deaths. Here are the counts (with partial 2017 data)

The first question you should ask is whether this is explained by population increases or by driving increases. That is, we want rates — deaths per unit of distance travelled

There’s still an increase, but now the 2017 partial data are in line with the increase. The increase cannot be explained simply by more cars being on the roads.

The next question is about uncertainty.  Traditionally, news stories about the road toll were based on one month of data and random variation could explain it all. We still need a model for how much random variation to expect.  What I said before was

The simplest mathematical model for counts is the Poisson process.  If dying in a car crash is independent for any two people in NZ, and the chance is small for any person (but not necessarily the same for different people) then number of deaths over any specified time period will follow a Poisson distribution.    The model cannot be exactly right — multiple fatalities would be much rarer if it were — but it is a good approximation, and any more detailed model would lead to more random variation in the road toll than the Poisson process does.

In that case I was arguing that there wasn’t any real evidence of a change, so using an underestimate of the random variation made my case harder. In this case I’m arguing the change is larger than random variation, so I need to make sure I don’t underestimate random variation.

What I did was fit a Bayesian model with two extra random components.  The first was the trend over time. To avoid making assumptions about the shape of the trend I just assumed that the difference between adjacent years was relatively small and random. The second random component was a difference between the trend value for a year and the ‘true’ rate for that year. On top of all of that, there’s Poisson variation.  Since the size of the two additional random components is estimated from the data, they will capture all the variation.

For each year, there is a 50% probability that the underlying rate is in the darker blue interval, and a 95% probability it’s in the light blue interval.  The trend is smoother than the data because the data has both the Poisson variation and the extra year-specific deviation. There’s more uncertainty in 2001 because we didn’t use pre-2001 data to tie it down at all, but that won’t affect the later half of the time period much.

It looks from the graph as though there was a minimum in 2013-14 and an increased rate since then.  One of the nice things about these Bayesian models is that you can easily and meaningfully ask for the probability that each year was the minimum. The probability is 54% for 2013 and 27% for 2014: there really was a minimum around then.

The probability that the rate is higher in 2017 than in 2013 is over 90%. This one isn’t just random variation, and it isn’t population increase.

Update: Peter Ellis, who has more experience with NZ official statistics and with Bayesian state-space time series models, gets qualitatively similar results

Q: Did you see “Just one extra banana or avocado a day could prevent heart attacks and stroke”

A: Hmm.

Q: It’s the potassium

A: Uhuh

Q: New Research Suggests

A: The effects of higher-potassium foods on blood pressure aren’t ‘new research’.  Look at what the American Heart Association says, or Harvard Health.

Q: Those sites don’t mention avocados, though. Is that what was new about the research?

A: No, that’s probably to meet the day’s quota for avocado stories.

Q: But at least the health message is real? They quote the researcher “The findings demonstrate the benefit of adequate potassium supplementation on prevention of vascular [hardening]”. With proper brackety things like we tell students to use.

A: We might prefer students to quote the rest of the clause , as the story does later: “demonstrate the benefit of adequate potassium supplementation on prevention of vascular calcification in atherosclerosis-prone mice” (emphasis added)

Q: So it probably wasn’t bananas, either.

A: No, high-cholesterol, high fat mouse food with high or low potassium.

Q: But the high-potassium mice lived longer? They had fewer heart attacks and strokes?

A: This is a lab experiment. It’s never going to end well for the mice. But they had stretchier arteries while they were alive.

Q: So what was the point, if we already knew higher-potassium diets with lots of fruit and veg are good for blood pressure?

A: The point was to find out how it works — which genes and proteins and so on.

From the world’s third-largest news agency:

1. The Nationalist Party?
2. National got 56 seats, not 58 — the graph seems to have the National results from the provisional count but the Labour and Green results from the final count
3. NZ First doesn’t use yellow
4. ACT, on the other hand, does.
5. But ACT is relatively unlikely to enter a left-wing coalition with Labour and the Greens

Team Ratings for Round 9

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 62 matches played, 43 of which were correctly predicted, a success rate of 69.4%.
Here are the predictions for last week’s games.

Predictions for Round 9

Here are the predictions for Round 9. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 14

The basic method is described on my Department home page.

After trying to deal with the Cheetahs playing their first team in the Pro14 and a second or third team in the Currie Cup, I have come up with the appropriate solution, which is to have two separate Cheetahs teams. Cheetahs is the team when there is no Pro14 game, so the first choice players should be playing. Cheetahs2 is the team when there is a Pro14 game that week, so the reserve players will be playing in the Currie Cup.

I arbitrarily started the Cheetahs2 at a rating of -30, which is a bit of a rough guess based on results so far, and reran all my predictions so far to produce this weeks predictions. Note that the Cheetahs2 rating has not changed very much over the games so far, nor has the rating for the Cheetahs.

I would welcome comments on the assumptions underlying this approach.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 39 matches played, 26 of which were correctly predicted, a success rate of 66.7%.
Here are the predictions for last week’s games.

Predictions for Round 14

Here are the predictions for Round 14. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

• Google has a camera that learns to take automatic videos of your kids and pets.  And they’ve gone to some lengths to make it less creepy than that sounds.

• It’s not just the inbuilt biases of the input data you have to worry about in statistical models/machine learning. People are also messing with the input deliberately.

• NY Times piece on personal genetic testing. (Disclaimer: I’m doing some consulting for a personal genomics company)

• Also from the Times, Jordan Ellenberg (mathematician) on “How Computers Turned Gerrymandering Into a Science“ (Now with a link!!)

• Where Americans get their science news and how much they trust various sources, from Pew Research.

• “The Subjects Planned for the 2020 Census and American Community Survey report released today inadvertently listed sexual orientation and gender identity as a proposed topic in the appendix,” the U.S. Census Bureau said in a statement to NBC News. “This topic is not being proposed to Congress for the 2020 Census or American Community Survey”

• ” It’s no longer good enough to shrug off (“briefly,” “for a small number of queries”) the problems in the system simply because it has computers in the decision loop.”

• Road deaths are up since 2013.  Contrary to what the NZTA spokesperson says, it can’t be explained by increases in cars on the road: there has been a change in the trend for deaths per unit distance travelled.

• Voting is now open for NZ Bird of the Year.  StatsChat doesn’t usually endorse bogus polls, but this one admits it’s just a publicity stunt.