Stats Chat

Team Ratings for Round 7

This year the predictions have been slightly changed with the help of a student, Joshua Dale. The home ground advantage now is different when both teams are from the same country to when the teams are from different countries. The basic method is described on my Department home page.

Here are the team ratings prior to Round 7, along with the ratings at the start of the season.

Performance So Far

So far there have been 35 matches played, 25 of which were correctly predicted, a success rate of 71.4%.

Here are the predictions for last week’s games.

Predictions for Round 7

Here are the predictions for Round 7. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

The Herald’s has two good stories recently (and other outlets are similar) about new data collection and analysis with smartphone apps.   If you read the stories carefully there’s been an outbreak of synecdoche in the newsroom — they are actually stories about new sensors that collect data, and smartphone apps that can be used to monitor the sensors — but that’s a minor detail.

As sensors based on electronics and micromechanics become cheap, it’s easy to collect more and more data.  This can be valuable — measuring blood glucose, or pollutants in the air — but it can also get out of hand.  Epidemiologist Hilda Bastian gives us this cartoon of the “Self-Stalker 5000” and writes about evidence-based monitoring in her blog at Scientific American

and Nick Alex Harrowell points out

The stereotype application could be defined as “bugging granny”. We’re going to check some metrics at intervals, stick them into a control chart, and then badger you about it.

To be fair, he’s worrying most about sensors that don’t send to a smartphone app, but many of the same principles apply.

“I wish there were more examples where data viz actually mattered. The case studies for us to lean on are sparser than they should be.”

Amanda Cox, NY Times chartmaker, interviewed at Harvard Business Review.  Includes a graph showing how the same unemployment report might be viewed by partisans of opposing parties.

The US non-profit journalism group Pro Publica has written a number of good stories recently about drug company payments to doctors.  They used the data that some US states force the drug companies to release.  They have a new story explaining why this isn’t as easy as it sounds, since there was no requirement that the data be released in any useful form.  For example, a lot of it was in PDF files.  As they write

Here’s how a PDF works, deep down: It positions text by placing each character at minutely precise coordinates in relation to the bottom-left corner of the page. It does something similar for other elements like images. A PDF knows about shapes, characters and their precise positions on the page. Even if a PDF looks like a spreadsheet — in fact, even when it’s made using Microsoft Excel — the PDF format doesn’t retain any sense of the “cells” that once contained the data.

They used a wide range of techniques: in some cases they could use the grid cells on the tables to work out which digits belonged to the same number, but in other cases they basically had to treat the PDF file as an image and use optical text recognition software on it, just as you would for a scanned bitmap. Most people wouldn’t go to these heroic lengths, and would rapidly decide to investigate other exciting stories.

Even Excel spreadsheets are only useful open data formats if they are structured so that it’s easy for a computer to find and extract the actual numbers from the worksheet. Stats NZ , who realise this, try to have data available both as Excel spreadsheets designed for visual display and in some useful downloadable form. Some other sources of NZ official data are not as helpful.

(via @adzebill on Twitter)

There’s been a lot of news recently about cold weather and snow in parts of the far Northern hemisphere that have  people living in them, especially English-speaking people.   As has typically happened with newsworthy cold snaps in recent years, this is balanced by unseasonably warm weather in parts of the far North that don’t have many  people living in them.

There are good reasons why the TV news doesn’t have much coverage of unseasonably warm weather in northern Greenland and the Arctic icecap. For a start, the local broadcasting infrastructure sucks.  It’s still important to remember that we only hear about weather in a fairly small fraction of the world.

The United States has surprisingly low social mobility: in every country, the children of the rich are more likely to be rich than the children of the poor, but the US is even worse than most Western countries.

Felix Salmon links to some graphs by Evan Soltas, looking at mobility in terms of education, with data from the US General Social Survey. He finds that people whose fathers did not go to university are much less likely to go to university themselves (unsurprising), and that this is true at all levels of income (more interesting).

I’ve repeated what Soltas did, but smoothing[1] the relationships to remove the visual noise, and also restricting to people aged 25-40 (rather than 18+)

In each panel, black is less than high school, dark red is high school, light brown is university or junior college and yellow is postgraduate. These are plotted by family income (in inflation-adjusted US dollars).  The left panel is for people whose fathers had at least a junior college degree; the right is those whose fathers didn’t.

The difference is striking, and as Soltas says, may imply a greater long-term value for encouraging education than people had thought.

[1] For people who want the technical details:  A sampling-weighted local-linear smoother using a Gaussian kernel with bandwidth $10000, ie, svysmooth() in the R survey package. Bandwidth chosen using the ‘Goldilocks’ method[2]

[2] What? $3000 is too wiggly, $30000 is too smooth, $10000 is just right.

Congratulations to David Farrar for his nomination last week in our Stat of the Week competition.

He nominated this table on the NZ Herald (click to enlarge the screenshot):

We liked that the Herald were quick to fix the mistake after being alerted of it, but it would have been helpful to also include a note that it had been updated since published.

Thomas Lumley commented that while the totals were just a mistake, not a premeditated misuse, the whole table was a bad idea.

Each week, we would like to invite readers of Stats Chat to submit nominations for our Stat of the Week competition and be in with the chance to win an iTunes voucher.

Here’s how it works:

• Anyone may add a comment on this post to nominate their Stat of the Week candidate before midday Friday March 29 2013.
• Statistics can be bad, exemplary or fascinating.
• The statistic must be in the NZ media during the period of March 23 – 29 2013 inclusive.
• Quote the statistic, when and where it was published and tell us why it should be our Stat of the Week.

Next Monday at midday we’ll announce the winner of this week’s Stat of the Week competition, and start a new one.

(more…)

These might perhaps be evidence for or against the previous post

• Keith Ng’s magic rotating donuts of what people do all day
• Worldwide inequality database on education
• Telegraph, on French wine regions
• Wired magazine on international organized crime (note: very slow to load)

Stephen Few writes a blog called “Visual Business Intelligence”, which if you know that “Business Intelligence” is a euphemism for “Data Analysis”, or “Statistics” is clearly in our field.

He has a recent post complaining about a new release of the “visual business intelligence” tool Tableau, in particular, its apparent enthusiasm for bubble charts and word clouds, and other things that don’t really work.  Early marketing, now removed, for this version actually used the heading “Crave more bling?” to describe the features.

As he points out in detail, bubble charts and word clouds are always and everywhere less informative than bar charts.  You should go read the whole thing. And perhaps his gallery of bad examples.

I learnt about Few’s post from a column (The Data Trail) in the Vancouver Sun. Under the headline In Defense of Eye Candy, Bling,and Tableau 8, Chad Skelton writes

There’s just one problem: bar charts are kind of boring.

A lot of people who create data visualizations — whether reporters, non-profits or governments — are fighting tooth and nail to get people to pay attention to the data they’re presenting in an online world crowded with endless distractions. And when you’re trying to make someone take notice  – especially if the subject is census data or transit figures — a little eye candy goes a long way.

Data visualizations aren’t just a way to present data. They’re often also the flashing billboard you need to get people to pay attention to the data in the first place.

He has a point, but this argument wouldn’t fly in other parts of journalism.  An editor would be unlikely to admit: “Yes, the story about numbers on benefit gave the wrong impression about blame, but a clear story explaining it was just the state of the economy would be boring”. When it comes to text or headlines, ‘tabloid journalism’ is an indictment, not a defense.

“Bling”, in particular,  is  perhaps an unintentionally honest term from the marketers. You wear bling primarily to prove you can afford it; you draw interactive packed bubble charts to prove you know how.

A more positive defense of complex infographics is that they function not as bling, but as art, and more importantly, as puzzles. As art, they are enjoyable to look at, but as puzzles, they are fun to explore. Andrew Gelman gives this example, by Michael Paukner (original here)

He notes

The headache is, I believe, part of the point. First, if the lines were direct you wouldn’t get the pretty Christmas tree pattern. Second, the investment required in following the lines makes you appreciate what you’ve learned. Third, the curvy lines are themselves a puzzle; as you trace them, you gradually learn the meaning of the y-axis.

It’s a familiar idea in education that you absorb information better if you have to do something discover it rather than just being fed it. If that’s why less-informative graphs and infographics are appreciated, perhaps we should be glad of evidence it isn’t only kids and scientists that still think it’s fun to find things out for themselves.

But if that’s the reason, it also warns of the limits of the strategy.  These displays are, actually, less efficient and accurate at conveying information.  In a situation where information does need to be conveyed efficiently and accurately, bling or eye candy wouldn’t matter so much, but puzzles need to be avoided.