Posts filed under Risk (185)

November 27, 2015

What should data use agreements look like?

After the news about Jarrod Gilbert being refused access to crime data, it’s worth looking at what data-use agreements should look like. I’m going to just consider agreements to use data for one’s own research — consulting projects and commissioned reports are different.

On Stuff, the police said

“Police reserves the right to discuss research findings with the academic if it misunderstands or misrepresents police data and information,” Evans said. 

Police could prevent further access to police resources if a researcher breached the agreement, he said. 

“Our priority is always to ensure that an appropriate balance is drawn between the privacy of individuals and academic freedom.

That would actually be reasonable if it only went that far: an organisation has confidential data, you get to see the data, they get to check whether you’ve reported anything that would breach their privacy restrictions. They can say “paragraph 2, on page 7, the street name together with the other information is identifying”, and you can agree or disagree, and potentially get an independent opinion from a mediator, ombudsman, arbitrator, or if it comes to that, a court.

The key here is that a breach of the agreement is objectively decidable and isn’t based on whether they like the conclusions. The problem comes with discretionary use of data. If the police have discretion about what analyses can be published, there’s no way to tell whether and to what extent they are misusing it. Even if they have only discretion about who can use the data, it’s hard to tell if they are using the implied threat of exclusion to persuade people to change results.

Medical statistics has a lot of experience with this sort of problem. That’s why the International Committee of Medical Journal Editors says, in their ‘conflict of interest’ recommendations

Authors should avoid entering in to agreements with study sponsors, both for-profit and non-profit, that interfere with authors’ access to all of the study’s data or that interfere with their ability to analyze and interpret the data and to prepare and publish manuscripts independently when and where they choose.

Under the ICMJE rules, I believe the sort of data-use restrictions we heard about for crime data would have to be disclosed as a conflict of interest.  The conflict wouldn’t necessarily lead to a paper being rejected, but it would be something for editors and reviewers to bear in mind as they looked at which results were presented and how they were interpreted.



November 25, 2015

Why we can’t trust crime analyses in New Zealand

Jarrod Gilbert has spent a lot of time hanging out with people in biker gangs.

That’s how he wrote his book, Patched, a history of gangs in New Zealand.  According to the Herald, it’s also the police’s rationale for not letting him have access to crime data. I don’t know whether it would be more charitable to the Police to accept that this is their real reason or not.

Conceivably, you might be concerned about access to these data for people with certain sorts of criminal connections. There might be ways to misuse the data, perhaps for some sort of scam on crime victims. No-one suggests that is  the sort of association with criminals that Dr Gilbert has.

It gets worse. According to Dr Gilbert, also writing in the Herald, the standard data access agreement for the data says police “retain the sole right to veto any findings from release.” Even drug companies don’t get away with those sorts of clauses nowadays.

To the extent these reports are true, we can’t entirely trust any analysis of New Zealand crime data that goes beyond what’s publicly available. There might be a lot of research that hasn’t been affected by censorship and threats to block future work, but we have no way of picking it out.

November 15, 2015

Out of how many?

Stuff has a story under the headline ACC statistics show New Zealand’s riskiest industries. They don’t. They show the industries with the largest numbers of claims.

To see why that’s a problem, consider instead the number of claims by broad ethnicity grouping: 135000 for European, 23100 for Māori, 10800 for Pacific peoples(via StatsNZ). There’s no way that European ethnicity gives you a hugely greater risk of occupational injury than Māori or Pacific workers have. The difference between these groups is basically just population size. The true risks go in the opposite direction: 89 claims per 1000 full-time equivalent workers of European ethnicities, 97 for Māori, and 106 for Pacific.

With just the total claims we can’t tell whether working in supermarkets and grocery stores is really much more dangerous than logging, as the story suggests. I’m dubious, but.

November 13, 2015

Blood pressure experiments

The two major US medical journals each published  a report this week about an experiment on healthy humans involving blood pressure.

One of these was a serious multi-year, multi-million-dollar clinical trial in over 9000 people, trying to refine the treatment of high blood pressure. The other looks like a borderline-ethical publicity stunt.  Guess which one ended up in Stuff.

In the experiment, 25 people were given an energy drink

We hypothesized that drinking a commercially available energy drink compared with a placebo drink increases blood pressure and heart rate in healthy adults at rest and in response to mental and physical stress (primary outcomes). Furthermore, we hypothesized that these hemodynamic changes are associated with sympathetic activation, which could predispose to increased cardiovascular risk (secondary outcomes).

The result was that consuming caffeine made blood pressure and heart rate go up for a short period,  and that levels of the hormone norepinephrine  in the blood also went up. Oh, and that consuming caffeine led to more caffeine in the bloodstream than consuming no caffeine.

The findings about blood pressure, heart rate, and norepinephrine are about as surprising as the finding about caffeine in the blood. If you do a Google search on “caffeine blood pressure”, the recommendation box at the top of the results is advice from the Mayo Clinic. It begins

Caffeine can cause a short, but dramatic increase in your blood pressure, even if you don’t have high blood pressure.

The Mayo Clinic, incidentally, is where the new experiment was done.

I looked at the PubMed research database for research on caffeine and blood pressure.  The oldest paper in English for which I could get full text was from 1981. It begins

Acute caffeine in subjects who do not normally ingest methylxanthines leads to increases in blood pressure, heart rate, plasma epinephrine, plasma norepinephrine, plasma renin activity, and urinary catecholamines.

This wasn’t news already in 1981.

Now, I don’t actually like energy drinks; I prefer my caffeine hot and bitter.  Since many energy drinks have as much caffeine as good coffee and some have almost as much sugar as apple juice, there’s probably some unsafe level of consumption, especially for kids.

What I don’t like is dressing this up as new science. The acute effects of caffeine on the cardiovascular system have been known for a long time. It seems strange to do a new human experiment just to demonstrate them again. In particular, it seems ethically dubious if you think these effects are dangerous enough to put out a press release about.


November 6, 2015

Failure to read small print


This story/ad/column hybrid thing on the Herald site is making a good point, that people don’t read the detailed terms and conditions of things. Of course, reading the terms and conditions of things before you agree is often infeasible — I have read the Auckland Transport HOP card T&Cs, but I don’t reread them to make sure they haven’t changed every time I agree to them by getting on a bus, and it’s not as if I have much choice, anyway.  When the small print is about large sums of money, reading it is probably more important.

The StatsChat-relevant aspect, though is the figure of $1000 per year for failing to read financial small print, which seemed strange. The quote:

Money Advice Service, a government-backed financial help centre in the UK, claims failure to read the small print is costing consumers an average of £428 (NZ$978) a year. It surveyed 2,000 consumers and found that only 84 per cent bothered to read the terms and conditions and, of those that did, only 17 per cent understood what they had read.

Here’s the press release (PDF) from Money Advice Service.  It surveyed 3000 people, and found that 84 per cent claimed they didn’t read the terms and conditions.

The survey asked people how much they believed misunderstanding financial terms in the last year had cost them. The average cost was £427.90.

So the figure is a bit fuzzier: it’s the average of what people reported believing they lost, which actually makes it more surprising. If you actually believed you, personally, were losing nearly a thousand dollars a year from not reading terms and conditions, wouldn’t you do something about it?

More importantly, it’s not failure to read the small print, it’s failure to understand it. The story claims only 17% of those who claimed to read the T&Cs thought they understood them — though I couldn’t find this number in the press release or on the Money Advice site, it is in the Mirror and, unsourced, in the Guardian.  The survey claims about a third misunderstood what ‘interest’  meant and of the 15% who had taken out a payday loan, more than half couldn’t explain what a ‘loan’ was, and one in five didn’t realise loans needed to be paid back.

As further evidence that either the survey is unreliable or that it isn’t a simple failure to read that’s the problem, there was very little variation between regions of the UK in how many people said they read the small print, but huge variation (£128-£1014in how much they said it cost them.

I’m not convinced we can trust this survey, but it’s not news that some people make unfortunate financial choices.  What would be useful is some idea of how often it’s really careless failure to read, how often it’s lack of basic education, how often it’s gotchas in the small print, and how often it’s taking out a loan you know is bad because the alternatives are worse.

November 3, 2015

Dogs and asthma

One News saysThe family dog or growing up on a farm could be the keys to reducing the chances of a young person suffering from asthma.

This is pretty good research. It’s obviously not a randomised experiment, but it uses the population administrative and medical data of Sweden to get a reasonable estimate of associations, and it is consistent with other population studies and has a reasonable explanation in immunology. One News gave all the relevant numbers, and got Dr Collin Brooks from Massey in as an expert. So that’s all good.

But (you knew there was a ‘but’), the population impact is smaller than the news story suggests.  That has to be the case: New Zealand, with very high asthma rates by international standards, already has fairly high dog ownership rates.  In fact, as often happens, this new study has found less benefit than earlier, smaller studies.

At current NZ asthma rates, for every extra 100 little kids who live with dogs, the research would predict that you’d prevent one or two cases of asthma. And that’s without worrying about, say, reduced housing options for households with pets.


November 1, 2015

Communicating swimming pool risks

There’s a story in the Herald tending to imply that the government is planning to repeal the laws that require fences around swimming pools, and that this will cost about 70 lives over the next ten years, for a saving of $17 million in `compliance costs.’  The story doesn’t quite say this, but it’s certainly possible for intelligent people to read it that way. Some did.

On the other hand, the Treasury Regulatory Impact Statement estimates the changes will save about six lives over the next ten years. It’s possible that the Treasury is that badly wrong but if they are it would have been helpful to see details of how and why.

Everyone agrees that fencing swimming pools saves lives. I mean, not just the Government, the Treasury, and Water Safety New Zealand, but everyone, even the Freakonomics guys whose whole shtick is not agreeing with things.  The question is what the law changes  do to pool fencing.

To start with, these changes do repeal the Fencing of Swimming Pools Act 1987, but not in the sense of making the requirements go away. They’ve been relocated as amendments to the Buildings Act 2004, with details in the Building Code.

The plans are

  • inspections reduced from three years to five years, but councils will be required to perform them (some now don’t inspect at all)
  • spa pools allowed to have child-proof covers instead of fences
  • putting fencing requirements in the building code, so compliance is more likely to happen automatically
  • an infringement notice as first penalty for violating the code, rather than the theoretical ability to prosecute immediately.

Water Safety New Zealand has said in the past that they think spa-pool covers are effective; at that time they were concerned that the covers wouldn’t be inspected (as quoted by Treasury from public comments to the 2013 version of the proposal).

It looks like the main real concern is the reduction in inspections both for councils who current inspect every three years, and from the omission of spa pools.

Requiring councils to inspect spa pools would be expensive (they estimate $67 million more over ten years). Three-yearly instead of five-yearly required inspections of swimming pools is estimated to cost an extra $19 million over ten years.  Those, not $17 million for the whole package, are the relevant cost-benefit numbers, relative to how many lives you think would be saved.

I could see three-yearly inspections being worthwhile if the rate of, say, broken swimming-pool gates in otherwise compliant fencing was high enough. I’d be a bit more surprised for spa pools, but it’s not inconceivable.  The story doesn’t say anything that could help me make up my mind.

There are also areas where the government could have tightened up rules but didn’t. For example, the Regulatory Impact Statement considered the option of forcing existing pools to comply with some building code changes, but estimated the cost would be about $10 million per life saved.

You can find the various government documents here, and Water Safety NZ’s submission here.

The story is encouraging people to make public submissions on the Bill. As a general principle, this is a good idea. In New Zealand, public submissions on legislation are, at least sometimes,  taken seriously. The chances will presumably be higher if you’re relatively precise about what you think should be changed in the bill and why — it’s not an opinion poll, it’s crowd-sourced review and editing.

October 22, 2015

Early NZ data visualisation

From the National Library of New Zealand, via Jolisa Gracewood


Types of motor-vehicle accidents in rural areas vary considerably from those ocourrlng In urban areas, as shown in tho above chart. Tho percentages are based on figures of the Transport Department in respect of accidents causing’ fatalltles during the twelve months, April I, 1932, to March 31, 1933.

The text goes on to say “The black section representing collisions with tram and train forms only I per cent, of the whole, through this type of accident appeals to the popular Imagination’ from its spectacular nature.”  Some things don’t change.

October 9, 2015

Predictive analytics and the rise of the machines

Some cautionary tales

  • “I would like to challenge this picture, and ask you to imagine data not as a pristine resource, but as a waste product, a bunch of radioactive, toxic sludge that we don’t know how to handle.” A talk by Maciej Ceglowski
  • How do you measure whether automated decision making ends up discriminating by race, when it doesn’t explicitly use race as an input? Two posts by Cathy O’Neil
  • A computer program that was accidentally trained to discriminate by gender and ethnicity
  • Why modern predictive analytics doesn’t give ‘algorithms’ in the sense of ‘recipes’, by Suresh Venkat (via @ndiakopoulos)
September 18, 2015

Compared to what? (transport chaos edition)

A while back, it looked as though the negotiations between NZ Bus and its drivers would break down and we would have bus strikes in Auckland. I considered various contingency plans: working from home for all or part of  a day, taking a train to Newmarket or Britomart and walking to the University, cycling, or catching a ride with a colleague who lives nearby. Some of these were options because we would have a week or so of warning before the strike.

If public transport in Auckland became permanently bad — if it went back to its state 20 years ago — I would have different options. I probably wouldn’t live in a house in Onehunga; I’d live in an apartment near the city centre. Moving to the city centre wouldn’t be a sensible response to a single day’s stoppage, but it would be sensible if the lack of buses was permanent.

Transport Blog has a post about the congestion benefits of the Wellington rail system, based on the week in June 2013 that it was taken out by a storm. On weekdays during this period, about 4000 people who would normally take the train into Wellington couldn’t. The roads became much more congested, and these delays can be valued (using plausible-looking assumptions) as worth over $5 million. Scaling this up to a full working year, the benefit to drivers in reduced driving time is worth rather a lot more than the public subsidy to the entire Wellington public transit system.

There’s a problem with simply scaling up the costs. If the Hutt Valley train line didn’t exist, some of those 4000 people would either live somewhere else or work somewhere else. Driving for an extra two hours each way was a rational response by them to a short-term outage, but in the long term they would reorganise their lives to not do it.

Now, there’s obviously a cost to moving from the Hutt to Wellington for these people — otherwise they’d be living in Wellington already — but the cost is less than would be estimated from the travel time during the outage. It’s hard to tell how much less without a lot more data and modelling.

On the other hand, while the storm data almost certainly overestimate the congestion-cost benefits of the train line, the magnitude of the estimated benefit is so large that the conclusion could quite easily hold even with better estimates.