January 23, 2020

Gender pay gaps

New Zealand and international media are reporting an new analysis of the gender pay gap among NZ academics. At one level this isn’t anything very surprising: there’s a gender pay gap, of the same percentage order of magnitude as in NZ as a whole (larger in Medicine, smaller in Arts).

As I’ve pointed out before, we know this is caused by gender, it’s not just some sort of correlation caused by confounding factors, since there aren’t any. What’s interesting is how it is that women come to be paid less. You could imagine a range of direct mechanisms:

  • slower promotion
  • lower pay at the same grade
  • less likely to be head of department/school
  • more likely to be at institutions where pay is lower
  • more likely to be in fields where pay is lower

And you could imagine possible factors leading into these

  • lower research ability
  • lower average age, because of past discrimination
  • interested in putting more effort into teaching or into service
  • pushed into putting more effort into teaching or into service
  • interested in putting more effort into childcare
  • pushed into putting more effort into childcare
  • discrimination in salary assignments
  • discrimination in promotion

and so on.

While many people have more or less informed opinions about these mechanisms, it’s often hard to get good data.  The research (by Associate Professors Ann Brower and Alex James of the University of Canterbury) takes advantage of the 2018 PBRF evaluations of NZ academics.  These evaluations were based on research portfolios selected to show the best research from each person (quality rather than quantity) and were evaluated by panels of NZ and overseas experts in each field.

In this paper, Brower and James got access to PBRF ratings and salary data for NZ academics, and so could look at whether women of similar age with similar PBRF scores had similar pay. As will surely astonish you, they didn’t.  In particular, it appears that women are less likely to be promoted to Associate Professor and Professor, with similar PBRF ratings, that men are.  Differences in age distribution and research performance explain about half the gender pay gap; the other half remains.

The big limitation of any analysis of this sort is the quality of the performance data.  If performance is measured poorly, then even if it really does completely explain the outcomes, it will look as if there’s a unexplained gap.  The point of this paper is that PBRF is quite a good measurement of research performance: assessed by scientists in each field, by panels convened with at least some attention to gender representation, using individual, up-to-date information.  If you believed that PBRF was pretty random and unreliable, you wouldn’t be impressed by these analyses: if PBRF scores don’t describe research performance well, they can’t explain its effect on pay and promotion well.

There could be bias in the other direction, too.  Suppose PBRF were biased in favour of men, and promotions were biased in favour of men in exactly the same way.  Adjusting for PBRF would then completely reproduce the bias in promotion, and make it look as if pay was completely fair.

Now, I’m potentially biased, since I was on a PBRF panel in 2013 (and since I got a good PBRF score), but I think PBRF is a fairly good assessment. I think the true residual pay gap could easily be quite a bit smaller or larger than this analysis estimates, but it’s as good as you’re likely to be able to do, and it certainly does not support the view that the pay gap is zero.

avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »

Comments

  • avatar
    Megan Pledger

    Interestingly “statistics” has the 5th lowest P(prof|Female). “Pure and Applied Mathematics” is 27th lowest (out of 42) and has triple the probability.

    To be fair, 95% confidence intervals must be pretty wide but, on the other hand, are confidence intervals warranted – there is no sampling error.

    4 years ago

    • avatar
      Thomas Lumley

      I’m glad to same there are now three more women professors of statistics than there were at PBRF time: congratulations Katrina Sharples, Rachel Fewster, Renate Meyer.

      I think confidence intervals are warranted — for the data generating process rather than a fixed population

      4 years ago

      • avatar
        Megan Pledger

        That would make P(prof|Female)=4/12 and put stats at the top of the pile, just ahead of law. I was surprised that P(prof|Female)=1/12.

        4 years ago

      • avatar
        Thomas Lumley

        Oh, and Elena Moltchanova at Canterbury is also newly Professorial.

        4 years ago

  • avatar
    Marcus Dewing

    Gender pay gaps are a serious matter, but I had to laugh when I read that the “PBRF is quite a good measurement of research performance”. Really?

    4 years ago

    • avatar
      Thomas Lumley

      Yes, actually. I mean, compared to the other ways people do it. But as I said, if you don’t think that, you shouldn’t put much weight on the estimates in the paper.

      I think PBRF is good enough for it to be informative that PBRF only explains half the pay gap. Reasonable people might disagree.

      4 years ago

      • avatar
        Marcus Dewing

        But — your opinion of PBRF is based on your specific area/panel. It sounds like this study uses data from all of the PBRF panels? I’m not sure you would say that “PBRF is quite a good measurement of research performance” if you were as familiar with what’s happening with other panels.

        4 years ago

        • avatar
          Thomas Lumley

          That’s why I explained how strengths or weaknesses of PBRF would affect interpretation of the results of the paper, so that people with different views didn’t need to agree with me.

          I don’t have time right now to program a sensitivity analysis, but maybe sometime in the next few weeks I might.

          4 years ago

    • avatar
      Ann Brower

      Hi, it’s great to see careful discussion of our paper. PBRF is, well, PBRF. As research scoring metrics go, it’s far more detailed than any other national exercise. I think the key assumption about PBRF in this paper is not that it’s perfect.
      The key assumption is, given its imperfections, it is likely to be equally imperfect for men as for women.
      Further, thinking like an employer … regardless of imperfections, PBRF is an important player in the NZ tertiary universe. So it’s reasonable to ask what role gender plays in how employers behave in the PBRF universe.

      4 years ago