Posts filed under General (3156)

Columbia University in New York is running an amazing-looking data journalism certificate called The Lede Program. The program director is Cathy O’Neill of mathbabe.org and Occupy Finance,  and the program advisor is Mark Hansen, statistician, computational scientist, and artist.

Anyway, their syllabus (and quite a bit of other content) is available on Github.

I’d like to quote a course outline by Cathy O’Neill

This course begins with the idea that computing tools are the products of human ingenuity and effort. They are never neutral and carry with them the biases of their designers and their design process. “Platform studies” is a new term used to describe investigations into these relationships between computing technologies and the creative or research products that they help to generate. How you understand how data, code, and algorithms affect creative practices can be an effective first step toward critical thinking about technology. 

 

Team Ratings for Round 12

The basic method is described on my Department home page. I have made some changes to the methodology this year, including shrinking the ratings between seasons.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 85 matches played, 46 of which were correctly predicted, a success rate of 54.1%.

Here are the predictions for last week’s games.

Predictions for Round 12

Here are the predictions for Round 12. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 16

The basic method is described on my Department home page. I have made some changes to the methodology this year, including shrinking the ratings between seasons.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 94 matches played, 62 of which were correctly predicted, a success rate of 66%.

Here are the predictions for last week’s games.

Predictions for Round 16

Here are the predictions for Round 16. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

In March, the US billionaire Warren Buffett offered a billion dollar prize to anyone who could predict all 63 ‘March Madness’ college basketball games. Unsurprisingly, many tried but no-one succeeded.

The New Zealand TAB are offering NZ$5 million to anyone who can predict all 64 games in the 2014 World Cup (soccer, in Rio de Janeiro (probably)). It’s free to enter. What’s it worth to an entrant, and what is the expected cost to the TAB?

If the pool games had equal probability of win/loss/draw and the finals series games were 50:50, which is the worst case for punters (well, almost), the chance of winning would be 1 in 5,227,573,613,485,916,806,405,226,496. That’s presumably also your chance of winning if you use random picks, which the TAB helpfully provides. At those odds, the value of an entry is approximately 1 ten-thousand-million-billionth of a cent (10^-19 cents), which is probably less than the cost to you of

By entering this Competition, an Entrant agrees to receive marketing and promotional material from the Promoter (including electronic material).

Of course, you could do better by picking carefully. Suppose that a dozen of the pool round games were completely predictable walkovers, the remaining 34 you could get  70% right, and you could get 50% for final games. That would be doing pretty well.  In that case the value of entering is hugely better — it’s almost a twentieth of a cent.   If you can get 70% accuracy for the final games as well, the value of entering would be nearly ten cents.

But if you can predict a dozen of the games with perfect accuracy and get 70% right for the rest, you’d be much better off just betting.  I looked at an online betting site, and the smallest payoffs I could find in the pool games were 2/9 for Brazil to beat Cameroon and 2/11 for Argentina to beat Iran.  If you have a dozen pool matches where you’re 100% certain, you can make rather more than ten cents even on a minimum bet.

So, what’s this all costing the TAB? It’s almost certainly less than the cost of sending a text message to every entrant, which is part of the process. There are maybe three million people eligible to enter, and a maximum of one entry per person. Given that duplicate winners will split the prize, I can’t really believe in an expected prize cost to TAB of more than 0.01 cents per entrant, which works out at about $1200 if every adult Kiwi enters. They should be able to insure against a win and pay not much more than this. The cost of advertising campaign will dwarf the prize costs.

The real incentive to enter is that there will be five $1000 consolation prizes for the best entries when no-one wins the big prize. What matters in figuring the odds for this  is not the total number of total entries (which might be a million), but the number of seriously competitive entries. That could be as low as a few tens of thousands, giving an expected value of entry as high as twenty cents if you’re prepared to put some effort into research.

[Update: It’s actually slightly worse than this, though not importantly so. You may need to predict numbers of goals scored in order to break ties when setting up the knockout rounds.]

Team Ratings for Round 15

The basic method is described on my Department home page. I have made some changes to the methodology this year, including shrinking the ratings between seasons.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 87 matches played, 56 of which were correctly predicted, a success rate of 64.4%.

Here are the predictions for last week’s games.

Predictions for Round 15

Here are the predictions for Round 15. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 11

The basic method is described on my Department home page. I have made some changes to the methodology this year, including shrinking the ratings between seasons.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 80 matches played, 44 of which were correctly predicted, a success rate of 55%.

Here are the predictions for last week’s games.

Predictions for Round 11

Here are the predictions for Round 11. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

May 20 is International Clinical Trials Day, commemorating James Lind’s trial of treatments for scurvy in 1747.

Our only Prime Minister, on Radio NZ

“I utterly reject those propositions. Twelve percent of households pay 76 percent of all net tax in New Zealand,” he said.

I’ve written about “net tax” before, both on StatsChat and elsewhere. It has to be defined and analysed carefully and non-intuitively in order to get these sorts of results.

Suppose we had an imaginary population divided into three groups. The ‘Low’ group, of 10 people, each pay $1000 in income tax, $1000 in GST, and receive $1500 in cash benefits. The “Middle” group, of 5 3 people, each pays $4000 in income tax, $3000 in GST, and receives no cash in benefits. The one person in the “High” group pays $17000 in income tax, $8000 in GST, and receives no cash in benefits.

According to Mr Key’s definition, the high-income group pays 71% of the “net tax”.  The middle-income group pays 50% of the “net tax”, and the low-income group pays -21% of the “net tax”.  That’s even though every person in this imaginary population pays more in tax than they receive in cash benefits.

There are three strange things going on here. The first is that GST is ignored. That’s obviously just wrong — GST is just as real as income tax.  The second is that cash benefits are treated differently from all other categories of government expenditure, even other categories such as subsidised medications that provide a direct, quantifiable individual benefit.  The third is that percentages behave strangely when you have a mixture of negative and positive numbers.  It’s quite possible, by choosing the subsets of the NZ population correctly, to find a group that pays well over 100% of the “net tax”.

Percentages become a lot less useful when they aren’t bounded by 100, and people who want to communicate accurately should avoid them in that situation.  And if you want to distinguish income tax revenue from GST revenue, you should clearly explain what you’re doing and why.

There’s a Stat-of-the-Week nomination for a Dominion Post article that I haven’t seen, because Stuff has had the good sense not to put it online. The press release is on Scoop, and from what our correspondent says, if you’ve read that, you’ve read the story. It’s about sex at the office, based on a ridiculously small sample selected from members of a dating website.  Since the dating website in question makes a lot of how different its members are from typical people, representativeness is not likely. Also, their infographic disagrees with the text of the release in at least one place.

That’s all standard. What’s interesting is the comparison of proportion of men and women who have had sex in various situations. Now, for the heterosexual majority, we have a basic accounting constraint in play. The office-sex survey says 20% of men and 3% of women have got it on in a conference room and 15% of men and 2% of women have done so in a storage room.   If these numbers were true there would be only three explanations: there are a lot more gay men around than other data suggest, and they really like the office; the few women who have sex at the office do so with many different men; or we have a Clintonesque definitional problem where the vast majority of the women involved don’t think what they did was sex.  More likely, it’s just evidence that the numbers are meaningless.

We’ve seen this problem before, but at least this is one problem the Herald’s story about holiday romance based on an Expedia press release avoided.

From the Herald

Housing Minister Nick Smith has revealed that Government held data on the proportion of New Zealand homes owned by offshore buyers, which he says is very low compared to other countries.

It turns out that the Government actually has data on the proportion of rental landlords who are overseas. Not the proportion of all homes, and not even the proportion of all rental homes. And even then, the proportion is based on whether the landlords are currently offshore, not whether they were offshore at the time of purchase, which is the topic of controversy (as the Herald does note).

It’s hard to do anything about the landlords vs rentals difference, but if the proportion of rentals owned offshore was also 11%, that would translate to about 4% of New Zealand homes, based on the home ownership figures from the Census.