Stats Chat

This is an expansion from the “Briefly” post about an NZTA summary of public comment on their SH1 Wellington proposals.

On Bluesky, @gwynebs had pointed out that some of the bars indicating levels of support didn’t appear to match the numbers attached to them — the “much better” category seemed inflated

A couple of days ago I noted there was a pattern to the distortion: it really was only the “much better” bar that was inflated and the other four were compressed in the same proportion. That is, some varying percentage was effectively being added to the “much better” level.  This is true for all five of the specific sections of the proposal,  but is not true for the two overall ratings in the middle of page 2, which appear correct. The bars are also correct in the much more detailed community engagement report; it’s just the summary that is wrong — which should indicate something about where things went wrong.

This is not rounding error. It’s much larger than that.

I went and measured the widths of all the bars in the five charts. These are in the same order as in the report: from top to bottom we have “2nd Terrace tunnel”, “Te Aro”, “Basin Reserve”, “2nd Mt Victoria tunnel”, and “Hataitai and Kilburnie”. The lower bar for each is cut from the NZTA summary. The upper bar has the correct percentages plus the necessary additional amount to make the bars line up — so the red is the amount that has been added to the “much better” category in the graph compared to the numbers. My bars and their bars don’t line up perfectly; that is probably rounding error. One possible explanation is that the red is some sort of “Don’t know” value that has inadvertently been put into the last bar — I could see that happening if the bars were drawn as pictures rather than as charts.

How much should we care about this? On the one hand, this sort of thing is probably corrosive to public trust in government data. On the other hand, this purports to be quantitative analysis of a self-selecting survey of the sort that attracts highly motivated and unrepresentative minorities*, so there’s a real limit to how seriously you should be taking the numbers.

Arguably, the point of this sort of survey is to see if there are surprising results — either something NZTA didn’t know about, or stronger opposition than they expected.  Even so, most people who aren’t the Advertising Standards Authority would think there’s something wrong with graphs that don’t match the data they purport to present.

*eg, people such as me

• For the day between March 31 and April 2nd, Andrew Gelman takes on an app that claims to find patterns in lotto numbers and make you money.
• RNZ reports the plans for tolls on the Road of Northland Significance, a charge of $4.50 each way from Warkworth to Te Hana (you will see some quotes of $14.20, which includes current tolls on the already-existing road to Puhoi). They don’t report what fraction of the cost the tolls will cover. Greater Auckland looked at the NZTA consultation papers about the tolling and say 35 years of tolling will raise $391m. That would be nearly 10% of the (phase 1) cost if you didn’t include interest; it’s a much smaller fraction when you do. And this is phase 1 — there are two more phases in the planned road to Whangārei.  Whether the road is worth the cost isn’t my specialty, but it’s a lot of cost.
• Len Cook (former Government Statistician) is in the Otago Daily Times disapproving of the planned removal of the census enumerations. We’ve covered this topic before.  The changes to the Data and Statistics Act are up for public comment, as are the necessary changes to the Electoral Act.   The electoral changes are not intrinsically controversial but are needed because electoral redistricting is currently triggered by the census. The electoral changes are important because they need a 75% supermajority in Parliament.
• RNZ reports on an NZTA report on public consultation about road changes in Wellington. First, the usual whinge: please link to this sort of report, so we can read it if your summary gets us interested!  Second, and the StatsChat motivation, the NZTA report displays pretty graphics of the public feedback, which are systematically wrong! For example, on the question “will a second Terrace Tunnel make things worse or better for you?” the lower bar is from the report and the upper bar is correct based on the percentages.  The right end of the bar is “better”, and is exaggerated
  
  Or the next question, about Te Aro improvements (original above, correct version below). Again, the “better” end is exaggerated
  
  I don’t think this is likely to be deliberate, but it’s a bad look

The Ministry of Transport have put up a fuel monitoring dashboard. It shows estimates of demand, supply, and price.

At the moment, the reduction in demand is less than 10%, a level of demand that’s probably not sustainable in the medium when global supply is down at least 25%. On the other hand, we are still at level 1 of the alert system, and even level 2 doesn’t ask for any real reductions in demand.

What this display doesn’t show is any sort of “time to running out”.  That’s probably sensible, because it’s not even well-defined, let alone predictable. If you define “running out” as some petrol stations being out of supplies then it’s already happened. If you define it as “no fuel in the country”, it probably won’t happen. And if you define it as level 3 or level 4 restrictions on supply then it’s a choice by the government based on unknown criteria, and so is hard to forecast statistically.

Team Ratings for Week 15

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 112 matches played, 77 of which were correctly predicted, a success rate of 68.8%.
Here are the predictions for last week’s games.

Predictions for Week 15

Here are the predictions for Week 15. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 21

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 140 matches played, 108 of which were correctly predicted, a success rate of 77.1%.
Here are the predictions for last week’s games.

Predictions for Round 21

Here are the predictions for Round 21. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Week 8

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 35 matches played, 23 of which were correctly predicted, a success rate of 65.7%.
Here are the predictions for last week’s games.

Predictions for Week 8

Here are the predictions for Week 8. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 13

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 60 matches played, 44 of which were correctly predicted, a success rate of 73.3%.
Here are the predictions for last week’s games.

Predictions for Round 13

Here are the predictions for Round 13. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 5

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 32 matches played, 19 of which were correctly predicted, a success rate of 59.4%.
Here are the predictions for last week’s games.

Predictions for Round 5

Here are the predictions for Round 5. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Week 5

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 28 matches played, 20 of which were correctly predicted, a success rate of 71.4%.
Here are the predictions for last week’s games.

Predictions for Week 5

Here are the predictions for Week 5. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

There have been two stories just recently in the Guardian about the dangers of opt-in surveys.  A survey from the respectable polling organisation YouGov reported a big increase in (Christian) church attendance among young people.  This was a bit of a  surprise, and didn’t seem to match up with other polling data (or with attendance counts by denominations that count attendance), but it was YouGov and it was what some people wanted to hear.

Apparently the problem was opt-in respondents.  This isn’t the completely useless opt-in clicky polls that our news sites put up from time to time; YouGov is a serious polling organisation.  However, I think it’s fair to say YouGov has tried to get accurate poll results by focusing more on statistical modelling of who responds and less on trying to get a good sample.  Again, that’s a perfectly reasonable strategy and has historically been competitive. You can’t get real random samples of people any more — not like in the 1950s — and so you get samples that are representative in some qualitative sense and reweight them to match the groups you’re trying to study.

You might think it’s strange that people would try to get into survey samples. It is strange, and that’s exactly the problem. Only a small fraction of people will try to get into surveys for the money, so those people are very unrepresentative, and while they are only a small fraction of the population that’s still a lot of people.   In the future, there’s the potential for LLM-based fake people to take surveys for the money (or just to be inconvenient), and they will be still worse.

When you start with a reasonably well-controlled sample and some people opt out, you have a subset of a reasonably well-controlled sample. It looks as though allowing too much self-selection can be qualitatively worse (though this is a one-off so far, and only provides limited evidence).

I also want to note that CNN reports a response from the Bible Society to the withdrawal of the survey report

The Bible Society said in a statement it was “deeply disappointed” by what had happened, but insisted the “wider picture” from other surveys pointed to “an increased engagement in faith among young adults compared to older generations.”

This isn’t a good reaction: the reason we found out the report was inaccurate was precisely that other evidence didn’t point the same way.