Stats Chat

The occasion is Nate Silver and the data-based predictions of the US election, but Mark Coddington raises a much more general point about the difference between ways of knowing things in journalism and science.

The journalistic norm of objectivity is more than just a careful neutrality or attempt to appear unbiased; for journalists, it’s the grounds on which they claim the authority to describe reality to us. And the authority of objectivity is rooted in a particular process.

But science finds things out differently, so journalists and scientists have difficulty communicating with each other.  In political journalism, the journalist gets access to insider information from multiple sources, cross-checks it, evaluates it for reliability, and tells us things we didn’t know.  In data-based journalism there aren’t inside secrets. Anyone could put together these information sources, and quite a few people did.  It doesn’t take any of the skills and judgment that journalists learn; it takes different skills and different sorts of judgment.

TL;DR: Political journalists are skeptical of Nate Silver because they don’t understand and don’t trust the means by which he knows what he knows. And they don’t understand it because it’s completely different from journalists have always known things, and how they’ve claimed authority to declare those things to the public.

There are new bus prediction boards at some stops (this one is in Newmarket), which show the scheduled time and the estimated time to arrival.

The most obvious benefit of these is that you can verify how the prediction system is unreasonably optimistic for buses still a long way off and then gets more realistic as they approach.  They also make clear how detached from reality the official punctuality statistics are.

Our only source of punctuality data for the 99%+ figures that keep getting reported is the companies themselves, and the definition of punctuality (the bus is no more than 5 minutes late at the start of the route and doesn’t get abducted by aliens on the way) is so useless that (pace Brian Rudman  and others) the companies would hardly need to lie about it.

Given that Auckland Transport thinks it knows where the buses are, it wouldn’t be too hard to switch to a meaningful definition: something like “proportion of specified timetable points where the bus is no more than 5 minutes late or 1 minute early”.  The figures would be lower, but that’s ok.  It’s hard to run buses to a tight schedule through a congested central city, and some of them will be late.  But you’re not going to get on-time arrival at bus stops by monitoring on-time departure at the start of the route.

I’d also note that the new bus prediction boards have missed a big opportunity.  According to the sign, the asterisk indicates that a bus is about to arrive.  Since the sign displays the scheduled time, they could have used the asterisk or some other symbol to indicate that the system didn’t currently have reliably location information for that bus.  It’s obvious that sometimes the system is using actual locations and sometimes it’s just relying on the timetable, and telling us what is going on would make it measurably less annoying when the time counts down to zero and no bus appears.

I’ve been following with interest the British Medical Journal‘s bold new move to refuse to publish research on drugs unless the clinical trial data is made available for independent analysis from January 2013. This is great news for other researchers, scientists and statisticians wanting to independently analyse and verify findings from studies.

In an editorial, the journal says they want to “leverage their power and publish only where there is a commitment to make the relevant anonymised patient level data available on reasonable request.” Further, they are campaigning for open data using Tamiflu as an important case study after being frustrated with years of trying to get drug company Roche to fulfill its promise to release full clinical trial results.

The BMJ quotes Ben Goldacre’s new book Bad Pharma:

“Drug companies around the world have produced some of the most amazing innovations of the past fifty years, saving lives on an epic scale. But that does not allow them to hide data, mislead doctors, and harm patients.”

Goldacre is one of a group of campaigners in the UK currently pushing the government to legislate for true transparency regarding clinical trials.

Watch a recent TED video from Ben Goldacre on the other “scandalous” problem of unreported trials of negative or inconclusive drug trial findings:

I believe Media Watch will be covering this story this weekend too.

Ben’s blog, Bad Science is always excellent reading, and I am looking forward to reading his newest book and hope that the BMJ’s campaign is successful.

XKCD:

Another random note on the US elections: the Congressional districts have been redrawn after the 2010 Census. In most states, this is done by the incumbent state government.   In a perfect world, the possible electoral advantages wouldn’t affect the boundaries.  As an example, look at the Pennsylvania map (especially districts 7 and 12)

Article II, Section 16 of the Pennsylvania Constitution says that the Commonwealth’s 50 senatorial districts and 203 representative districts “shall be composed of compact and contiguous territory as nearly equal in population as practicable.” It also says that “Unless absolutely necessary, no county, city, incorporated town, borough township or ward shall be divided in forming either a senatorial or representative district.”   

• Stuff has a story about real estate that goes beyond nominal prices and has meaningful comparisons with the past (based on ratios to average household disposable income)
• If you want somewhere to obsessively follow the US election, there’s live blogging at fivethirtyeight.com
• The Washington Post looks at Google trends for the query “who is running for president?”
• Stuff has a story about studies of multivitamins, reporting that they didn’t find the health benefits they were looking for, and then concluding “The bottom line: Dietary supplements have varied effects, and whether one is right for you may depend on your personal health profile, diet and lifestyle.” You could have said that (more convincingly) without the research

An interactive graphic from the New York Times, using data from the American Time Use Survey, where a representative sample of people is phoned up and asked what they did the previous day.

Compare the NY Times graphic to the charts produced by the Bureau of Labor Statistics, which are much less fun to explore, but do present some useful comparisons in simple formats.

That’s lead, Pb, the element. The Herald has a headline “Dangerous lead levels in Aussie kids – experts”. The article says

Health experts are calling for new national guidelines amid fears that up to 100,000 young Australian children could have dangerously high levels of lead in their blood.

The new proposed law for ‘natural remedies’ will provide some limits on health claims: they must be supported by scientific evidence or traditional evidence.  It’s depressing that this has to be considered an improvement over the current situation.  It’s also unfortunate that there is a special exemption for the largest category of health claims that have support neither from tradition nor science: homeopathic remedies (invented in 1796).

If ‘traditional’ evidence required a tradition of use of the product in essentially the form being sold and for essentially the indication being marketed, I wouldn’t object.  So, for example, it’s reasonable to say that ginger has been used traditionally in China to stop coughing, because it has. Many people would consider this sufficient reason to buy and use ginger for coughs, and I don’t see why there’s a vital national interest in deterring them.  What should be stopped is the ability to claim that a treatment is known to work simply because it has been traditionally used, and the ability to make up new ‘traditional’ uses for marketing convenience.

In the end, the impact of the law will depend on regulations and enforcement, not on statute.

Requiring ‘scientific evidence’ won’t get us very far without standards for the level of evidence required.  As StatsChat readers know, a huge variety of plant extracts have been shown to kill cancer cells in test tubes, or affect biochemical measurements in cell cultures. Newspapers are willing to take this as sufficient basis for stories and headlines claiming ‘can prevent cancer’ or ‘is helpful in treating diabetes’. If the proposed Authority takes a similar view, almost anything will be allowed.

Enforcement is also critical.  In theory, the US requires vendors to have scientific evidence for claimed health effects of natural remedies, but in practice they can get away with almost anything, because the FDA doesn’t have the resources for enforcement and there’s no practical way for private citizens to act. At the moment, the NZ Advertising Standards Authority has the ability to remove ads making health claims for which the vendor cannot supply evidence, and they have the willingness to act on public complaints from anyone who is unhappy about misleading health claims. The new law’s requirement for prior notification of health claims could strengthen enforcement, or weaken it, depending on how the requirement for evidence is interpreted.

Each week, we would like to invite readers of Stats Chat to submit nominations for our Stat of the Week competition and be in with the chance to win an iTunes voucher.

Here’s how it works:

• Anyone may add a comment on this post to nominate their Stat of the Week candidate before midday Friday November 9 2012.
• Statistics can be bad, exemplary or fascinating.
• The statistic must be in the NZ media during the period of November 3 – 9 2012 inclusive.
• Quote the statistic, when and where it was published and tell us why it should be our Stat of the Week.

Next Monday at midday we’ll announce the winner of this week’s Stat of the Week competition, and start a new one.

(more…)