Posts filed under General (3152)

• Animated commuting map for every county in the US. Something like this could be done with StatsNZ data….

• Alex Harrowell at The Yorkshire Ranter writes about two new papers combining large-scale data mining with sampling and human interpretation to study China’s ’50-cent party’ (五毛党) internet commentators

• Canada Post have given up their lawsuit against a group that crowdsourced Canadian postal code data. (<looks pointedly in direction of NZ Fire Service>)

• In the US, increasing numbers of households have smartphones and don’t even have landlines for broadband internet.

• Y-axes, from the UK Office of National Statistics

Kareem Carr, guest-posting at Mathbabe.org

All three principles have one underlying idea. Bad data science obscures and ignores the real world performance of its algorithms. It relies on little to no validation. When it does perform validation, it relies on canned approaches to validation. It doesn’t critically examine instances of bad performance with an eye towards trying to understand how and why these failures occur. It doesn’t make the nature of these failures widely known so consumers of these algorithms can deploy them with discernment and sophistication.

• “David Cameron should ban hedge funds from trying to cash in on the EU referendum by commissioning private exit polls to speculate on sterling before the official result, Labour’s deputy leader has said.” (Guardian) But. “If you think that this is bad — and Watson probably isn’t alone in thinking that it’s bad — then it seems to me that you have to identify which part is bad. Is it asking someone how she voted? Is it asking lots of people how they voted? Is it making a prediction about the Brexit vote? Is it trading based on your prediction? Which specific thing would you make illegal?” (Matt Levine)

• Generation Zero likes trains, and thinks other people also like trains. Rather than just asserting this or putting up a petition, they’re trying to crowdfund a real opinion poll to find out Auckland public opinion on maintaining a train option for the proposed harbour crossing. Obviously they’re doing this because they think they know what the answer will be, but it’s still a welcome step towards evidence-based lobbying.

• Information visualisation for concert-goers: a graph of the symphony you’re listening to, from the Toronto Symphony Orchestra

• Rain pattern graphics: San Francisco Bay Area, Hong Kong

• Google’s ‘Digital Ethicist’ on how software design hijacks people’s minds — changing the (implied) question to affect people’s decisions.

• Looking at AirBnb vs actual housing availability and price in major cities across the US

• Who marries whom (by occupational category)? The graph is pretty hard to use, but it’s still interesting. Sadly, ‘statisticians’ are lumped in with ‘mathematicians’.  (via Andrew Gelman)

I didn’t see this Stuff story at the time, but it was discussed on Twitter by Tess McClure (@tessairini).

The 2016 Global Slavery Index examines practices such as forced labour, human trafficking, child exploitation and forced marriage, surveying 43,000 people in 25 countries.

The number of people living in slavery in New Zealand has increased from 600 in the 2014 Global Slavery Index.

New Zealand and Australia have the lowest level of slavery prevalence in the Asia Pacific region with an estimated 0.018 per cent of the population in modern slavery.

If you survey 43,000 people in 25 countries you won’t be surveying very many in New Zealand, so where did this number come from?  The story doesn’t give any more details, but @tessairini found a ‘detailed methodology’ paper (PDF).

They didn’t survey any people in New Zealand. Or in Australia. Nor had they in 2014.

The survey part of the research is pretty much irrelevant to the estimates for New Zealand. The methodology paper describes another approach that

…can be applied in countries where nationally representative random sample surveys will not necessarily work. This is particularly the case in more ‘developed’ countries, where low levels of vulnerability mean that there are few cases to report, where law enforcement is strong and organized crime is more hidden, and where the resulting numbers are so small, that even if they were not hidden, they would be highly unlikely to be found and selected for interview in a random sample survey.

For the UK and the Netherlands the survey used data from the overlap of multiple lists. The UK estimate is described in Significance magazine, the popular-audience publication of the Royal Statistical Society. In all, 2744 victims of human trafficking were identified in the UK, from a total of six sources, so it’s possible to look at how many of these people were missed by each source, and estimate how many more might have been missed completely. The estimated total is between 10,000 and 13,000.

So, there’s survey data for 25 countries not including New Zealand or Australia, and multiple-list data for two further countries not including Australia and New Zealand.  We still haven’t found out where the New Zealand estimate comes from.

The final step is extrapolation from measured countries to unmeasured countries. The researchers measured a whole lot of variables that might be relevant, and divided the countries into groups that looked similar. They then applied the frequencies from the measured countries in each group, with a few adjustments, to the unmeasured countries.  If you look at the data in the Stuff story, Australia and New Zealand have the same estimated prevalence of slavery, 0.018% of the population. That’s also essentially the same as the UK estimate, so presumably we’re in the same group as the UK and that’s where the real data come from.

If you want a global estimate of the number of people affected by slavery, this is a perfectly reasonable approach. It’s probably kind of ok as an estimate for the number in New Zealand. On the other hand, the index data doesn’t support claims of change from year to year in New Zealand, and it doesn’t say anything about particular risks.

It makes sense to get local experts to talk about the industries and practices that might cause problems in New Zealand, as Stuff did, and what can be done about them, but the index estimate is just that New Zealand is about the same as the UK.

How did the statistical programming language R grow from a simple help-out for undergrad students to a global sensation? Associate Professor Ross Ihaka (right) of the University of Auckland tells the story in the latest issue of alumni magazine Ingenio.

And … here is some niceness from a fan who has read the story today. Thanks, Mike!

I have updated these predictions because I had not realised that the Chiefs Crusaders game was on a neutral ground.

Team Ratings for Round 15

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 108 matches played, 79 of which were correctly predicted, a success rate of 73.1%.
Here are the predictions for last week’s games.

Predictions for Round 15

Here are the predictions for Round 15. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 13

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 92 matches played, 53 of which were correctly predicted, a success rate of 57.6%.
Here are the predictions for last week’s games.

Predictions for Round 13

Here are the predictions for Round 13. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

For Wednesday’s quiz:

Remaining life expectancy for NZ men gets down to five years between ages 85 and 90. That’s true for men and women, and for Māori and for non-Māori, and for Pasifika.  (tables here) — the differences in life expectancy at birth have gone away by that age.

From the Herald, apparently original (or at least ahead of the UK)

A study from the United Kingdom has found that a glass of beer contains “significantly less” sugar than a can of Coke, a cappuccino or a glass of cordial.

The Campden BRI Food and Innovation study analysed the calorie content of 52 alcoholic drinks and found that most beers have less than 1 gram of sugar per 100ml.

That is, of course, true.  Beer tends to have low sugar because the yeast eats it all.  If anyone’s main concern about the health impact of beer consumption is the sugar, they are indeed worrying about the wrong thing.

Beer does have some other sweet-tasting carbohydrates that are digestible by people, though not by yeast.  More importantly, though, beer has alcohol, and that’s where most of its energy content comes from.

A discussion of sugar in beer, including detailed numbers, focused on health and whether it “makes you fat”, which fails to mention alcohol or total energy content, doesn’t happen by accident.  You’ve got to respect the publicist’s skills.

Q: Did you see there’s a new hangover cure? An ice-cream?

A: Really?

Q: Well, the Herald and Reuters think so. How does it work?

A: I think that question assumes facts not in evidence.

Q: How do they say it works, then?

A: It’s got 0.7% raisin tree fruit juice.

Q: That … doesn’t sound like very much.

A: About half a gram, probably. One Panadol has half a gram of paracetamol, so they’re claiming pretty high potency.

Q: But don’t they say it works in rats? There’s a journal and science and stuff! How much did the rats get?

A: They don’t link to the paper. Or give researcher names.

Q: Well, of course not. But you can find it, right?

A: I expect so.

<crickets>

A: Ok. That was relatively annoying.

Q: Didn’t you just need to search for ‘raisin tree juice neuroscience’ or something.

A: No, that gets you copies of this story.

Q: What about on PubMed, where you usually find things?

A: The paper turns out not to have “raisin tree juice” in the title. Eventually I found out the botanical name of the plant and searched for that at the journal website, which has full-text search.

Q: <world’s tiniest violin> How much juice did the rats get?

A: They didn’t get juice. They got a single chemical extracted from the juice, dihydromyricetin, injected.

Q: The ice-creams aren’t going to be injected, though?

A: No, I’m glad to say.

Q: What would be a comparable human dose?

A: They based the rat dose on traditional doses of the dried fruit in humans: “Clinically, the Hovenia dosage range used for hangover is 100–650mg/kg”

Q: So in a hypothetical 70kg pers0n?

A: Works out as 7g-45g. And that’s dry, without the extra water in the juice form.

Q: At least 14 ice-creams. How long did it take the rats’ hangovers to go away after being injected?

A: The rats got the chemical at the same time as the alcohol, so the idea was the hangover was being prevented, perhaps by blocking effects of alcohol in the brain so the rats didn’t really get as drunk.

Q: But if it works by not getting drunk there are cheaper ways.

A: Indeed.