Posts from March 2012 (64)

Statistics New Zealand has been undertaking the task of digitising their Yearbooks, dating back to 1893. The digitisation is not simple scanning which would not allow the easy copying of data.

One method of obtaining data which I have tested and appears to work satisfactorily is to use Excel 2010. Go to the Data tab, and select From Web. In the address box give the address of the page containing the table you wish to import. (For example from the 1893 Yearbook, I chose the page with the address http://www3.stats.govt.nz/New_Zealand_Official_Yearbooks/1893/NZOYB_1893.html#id333C618.) Then go to the table you want, click on the Yellow Arrow as directed, and you have your table in Excel.

I was alerted to this project by the interview last Saturday by Kim Hill of Claire Stent from Statistics New Zealand. You can get a podcast or listen to this by going to http://www.radionz.co.nz/national/programmes/saturday/20120324.

I would nominate this project for Stat of the Week, but I don’t think I am eligible.

A popular decision rule for investment and other forms of gambling is the Kelly criterion, named after mathematician (and successful investor) John Kelly.  In the long run, following this rule will maximise long-run expected wealth.

If we assume that the story in Stuff about Wairarapa bettors having had a 2:1 return in the past year can be applied to tomorrow’s Powerball (which it can’t), we can look at what that would imply about rational betting.

The Kelly criterion specifies what fraction of your total wealth you should spend on an investment opportunity.  The fraction is always less than your probability of winning.  With 2:1 expected payoff and large odds, the recommended fraction is about half the probability of winning

The chance of the top Powerball prize (since this isn’t a ‘must win’ week) is 1 in 38 million for a $1 bet, so you should bet less than 1 dollar for each 76 million dollars of your current disposable wealth.   For most of us, that’s less than one dollar.

It’s worth noting that while not everyone supports the Kelly criterion, most of the critics suggest that you should bet less than the criterion recommends, not more.

(via a commenter at Cornell physics blog The Virtuosi)

Recently, I mentioned the possibility of using bus timing data to probe congestion on Auckland roads.  This idea has been bypassed by Google, who now provide real-time congestion maps of New Zealand using smartphone location data.

If you run Google Maps or Google Navigation, you have the option of sending anonymous GPS-based location data to Google, so they know the locations of lots of phones.  By tracking the speed of phones that are moving along roads, they can work out the traffic speed, and measure congestion.   This is harder than it sounds — GPS accuracy on its own is not enough to distinguish phones in cars from phones carried by pedestrians — but using combined location and speed data they can even give separate congestion information in each direction on many roads.

For example, if you were coming to our public lecture on Tuesday, you might look on Google Maps and click on the “Traffic” label, and see that Symonds St is totally clogged, and decide to come up Grafton Rd instead.

Team Ratings for Round 5

Here are the team ratings prior to Round 5, along with the ratings at the start of the season. I have created a brief description of the method I use for predicting rugby games. Go to my Department home page to see this.

Performance So Far

So far there have been 32 matches played, 16 of which were correctly predicted, a success rate of 50%.

Here are the predictions for last week’s games.

Predictions for Round 5

Here are the predictions for Round 5

Team Ratings for Week 6

Here are the team ratings prior to Week 6, along with the ratings at the start of the season. I have created a brief description of the method I use for predicting rugby games. Go to my Department home page to see this.

Performance So Far

So far there have been 34 matches played, 21 of which were correctly predicted, a success rate of 61.8%.

Here are the predictions for last week’s games.

Predictions for Week 6

Here are the predictions for Week 6. The prediction is my estimated points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

As my good friend and colleague Thomas Lumley points out we have plenty of Lotto-based silliness to tide us over until the next stupid health related press release from a conference with no quality checks. Case-in-point is the article Powerball could be in the stars in Thursday’s NZ Herald (29 March, 2012, A5): (also nominated by Sammie Jia for Stat of the Week).

The article reports the frequency of zodiac signs from a survey of 104 first division Lotto winners, and gleefully touts Taurus as the luckiest star sign with 13% of the total. The article gives us a summary table:

Of course, all keen Statschat readers will note that this table does not add up to 100%, nor does it show all twelve zodiac signs, which is not very helpful. Buried in the text is the additional information that Aries and Cancer combined make up 4% of the total.

If we spread the remaining probability over Gemini, Sagittarius and Scorpio, and make the not entirely justified assumption that the distribution of zodiac signs is uniform (which is exactly what the NZ Herald has done), then we can perform a simple chi-squared test of uniformity. This yields a P-value of 0.22, which for most frequentists isn’t exactly compelling evidence.

Being a Bayesian, I prefer to assume multinomial sampling with the prior on the probability of success being uniform. The figure below shows posterior credible intervals (based on 10,000 samples) for the true probability of success. The red dots are the observed values. The dashed line is the equal probability line (0.083 = 1/12).

All of the intervals overlap confirming our statistical intuition that all we are really observing is sampling variation. Yes, Ares and Cancer do fall below the line, but they are not significantly different from the other signs. You can, of course, not believe me – in which case Thomas has some tickets from last week’s draw going very cheap and your chance of winning is almost the same.

A flowing wind map of the USA, from hint.fm, who also have other beautiful infographics. Click for the live version

The Herald has managed to top Stuff’s suggestion that Powerball is in Wairarapa with a story saying it’s “in the stars”.

Of course, they don’t mean the actual stars, they mean European-style astrological birth signs.  Allegedly (and it’s hardly the sort of claim that it’s worth verifying),  15% of winners over the past two years were born under Taurus.   Now, since there are 104 winners and 12 astrological signs, it’s arithmetically unavoidable that some signs will get more winners than others, but how many more would you expect in a typical two-year period?

The easiest way to approximate this is to ignore the fact that slightly more births occur in some months and just sample 104 winners equally from 12 groups.  Repeating this 10,000 times takes a few seconds, and we get a table showing the percentage chance that the most-sampled star sign will have at least 10, 11, 12, … winners.

Getting 15% in one group is not at all  surprising.

This still overestimates the surprise: if there hadn’t been an overall pattern, the story could have said that the chance of winning depends on your star sign and on the time of year, for example.    In fact, they did this as well: the second part of the story says that Taurus is lucky now, but that Capricorn will be lucky in a  few weeks.   Of course, the fact that the second part contradicts the first part isn’t mentioned anywhere, and no data are presented to gesture in the direction of evidence for the claim.

We’re very good at seeing patterns, even when they don’t exist, and statistics is one way of ameliorating this problem.

So what is the best time to buy a lottery ticket?  As Scott Adams shows, the day after the drawing — they are a lot cheaper then.

There’s recently been a lot of publicity about the views on internet congestion of a visiting Brit.  Next Tuesday, in Auckland, there’s a public lecture on internet congestion by a different visiting Brit, one who actually knows something about the topic.   Frank Kelly is a Professor of Mathematics at Cambridge, and a Fellow of the Royal Society.  His research is on the design and control of  networks: both abstract ones and concrete ones such as the Internet and the traffic system.

Professor Kelly is visiting the University of Auckland to work with researchers here, and will kick off his visit with the public lecture:

The Internet has attracted the attention of many theoreticians, eager to understand the remarkable success of this diverse and complex artefact. One strand of this effort has been a framework that allows the various detailed algorithms used to control congestion, choose routes and allocate resources to be seen as a distributed mechanism solving a global optimization problem. The talk will review the framework, and discuss topics such as fairness and stability, as well as current engineering efforts to improve the reliability and robustness of the Internet.

Stuff has an interesting story on self-publishing and how e-books and Amazon make it easier.  I’m in favor of anything that produces more good books, but I would like to give two warning notes, one statistical and from my experience, and one non-statistical and from other people’s experience.

The statistical warning note is that most books on Amazon don’t sell much, if at all.  I have a book, a graduate text in a minor area of statistics, for which Amazon gives me sales data.  This book sells 2-4 copies a week through all North American bookstores (and so less than that just through Amazon). Even so, it has never been out of the top 10% of Amazon’s sales rank, it usually is at about the 95th percentile, and sometimes makes it into the top 1%.   Nearly all books on Amazon sell less than this.

The non-statistical warning note is that the self-publishing industry is historically noted for having a lot of scams.  A bit of Google due diligence is useful before you spend money on things.