Posts filed under General (3156)

The “It’s not paranoia if..” issue

• A new initiative, Data Justice, concerned with widespread commercial data collection and analysis as a threat to privacy and equality.

• Some of the reasons that data-based decision making goes bad

• Trying to get “open” data in New Jersey: “initially refused to answer The Jersey Journal’s OPRA request because it didn’t make it on the agency’s standardized OPRA form, which wasn’t available on the NBHA website. Even after a reporter noted that in 2009 the state Supreme Court ruled standardized forms aren’t necessary, Earl wouldn’t accept a request on anything but the agency’s form.”

• Reidentification of ‘anonymised’ data: ‘When data is released after applying current ad-hoc de-identification methods, the privacy risks of re-identification are not just unknown but unknowable.’

I posted a version of this graph with ten years of weekly data, and Mark Stockdale pointed out there are quarterly data back to 1983 (isn’t official data wonderful?). You’ll need to click the graph to embiggen for easy viewing.

The horizontal axis is the import cost plus freight and insurance (with CPI adjustments to 2013 NZ dollars), and the vertical axis is the importer margin, which covers transport and sale costs within New Zealand, and profit. The idea is that local costs are typically slowly varying, so that short-term variation in margin tracks short-term variation in profit. The label for each year is on the data point for June.

The import cost plummeted in the early 1980s, soon followed by a drop in the importer margin. That’s presumably Rogernomics and its consequences. The cost stayed fairly stable and low in the 1990s and the margin drifted down.  Then the cost increased after 1999, with the margin staying stable. We’ve recently entered a new pattern, with margin drifting upwards.

A final note: the import cost is about the same as in 1983, and so is the retail price (in real terms). The reduction in importer margin since 1983 has been almost exactly matched by an increase in taxes, though the taxes would probably be higher under a realistic world carbon price.

• Large-scale data cleaning: the US Social Security Administration has social security records but no death records for 6.5 million people over 112, ie, about 6.5 million more than the number of people over 112 in the world. Nearly 4000 of these people are trying to get jobs “During Calendar Years 2008 through 2011, employers made 4,024 E-Verify inquiries using 3,873 SSNs belonging to numberholders born before June 16, 1901.”

• Data mining to see whether an ordinary taxi or Uber is cheaper for a specific trip — so far just in New York, and ignoring surge pricing.

• First FDA approval of a ‘biosimilar’ drug — the analogue of ‘generic’ for biologicals. Copying a biologic treatment  such as a protein hormone or an antibody is much harder than copying a small molecule (where the patent gives the necessary details), so the makers can charge more for it: in this case, only a 30% discount relative to the brand-name version. Biosimilars will be an important issue for Pharmac in the future: its second and third biggest medication expenses are for two biologicals.

• Census at School (or, in this context, Tatauranga Ki Te Kura) was on Māori TV’s news programme Te Kāea yesterday, with StatsChat contributor Julie Middleton explaining.

Team Ratings for Round 3

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 16 matches played, 10 of which were correctly predicted, a success rate of 62.5%.

Here are the predictions for last week’s games.

Predictions for Round 3

Here are the predictions for Round 3. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 6

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 34 matches played, 21 of which were correctly predicted, a success rate of 61.8%.

Here are the predictions for last week’s games.

Predictions for Round 6

Here are the predictions for Round 6. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

• There will be SCIENCE at the Auckland Festival on Saturday: Dr Michelle ‘Nanogirl’ Dickinson blowing things up, Dr Siouxsie Wiles (and artists, and you) lighting things up, and panel discussions.

• ‘In the 17th century, another genre of paintings emerged, showing public administrators holding their books open for all to see. More than 100 of these paintings were produced between 1600 and 1800. Transparency became a cultural ideal worthy of art.’ Jacob Soll writing in the Boston Globe about the financial data revolution of the 16th century.

• “The next big milestone for the project is to get a judge to rule in favor of a tenant based on Heat Seek data. That would set a precedent that the courts see these devices as reliable and unbiased evidence.” New York, like many US cities, has temperature standards for apartments where the landlord controls the heating system. Heat Seek wants to provide independent data using internet-connected thermometers.

• Does the popularity of party leaders affect voting? In the UK it seems the answer is “sometimes, a bit”.  (via Alex Harrowell)

There are probably some StatChat readers who don’t read the NZ Herald, so I’ll point out that I have a post on the data blog about election donations.

Team Ratings for Round 2

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 8 matches played, 5 of which were correctly predicted, a success rate of 62.5%.

Here are the predictions for last week’s games.

Predictions for Round 2

Here are the predictions for Round 2. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 5

The basic method is described on my Department home page.

Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 27 matches played, 17 of which were correctly predicted, a success rate of 63%.

Here are the predictions for last week’s games.

Predictions for Round 5

Here are the predictions for Round 5. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

• “Yesterday the Herald reported on some confusion about what’s happening to New Zealand property prices. QV said prices were continuing their seemingly inexorable rise, while Barfoot & Thompson said that in Auckland prices had dipped, based on its own sales.” Aaron Schiff, arguing that housing price data should be open
• Another tea story, but this one does say what experiments were done and what the equivalent dose would be in humans.
• Predictive models in language: autocompleting to ‘Qatar’
• Predictive models in language: detecting ‘fake’ (or Gaelic) names on Facebook 
• Predictive vs causal explanations in language, from XKCD