Goodhart's law

A school library runs a reading challenge: the class that reads the most books by spring wins a pizza party. Book counts triple. The librarian notices that the most-borrowed shelf is now the thinnest picture books, and a later survey finds students aren’t reading for any longer than before.

Before the challenge, the number of books a student read was a rough sign of how much they read. The prize made the count itself the goal, and the cheapest way to raise a count is to choose shorter books. Every number reported was accurate, and nobody broke a rule. The count simply stopped measuring what it had been chosen to measure.

The economist Charles Goodhart made the point in a 1975 paper on monetary policy in the United Kingdom, about statistical relationships that policymakers were relying on for control. In the form usually quoted: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” A 2021 editorial in the Journal of Graduate Medical Education describes the remark as initially “a jocular aside”.

The version most people know is not Goodhart’s wording. It comes from the anthropologist Marilyn Strathern, writing in 1997 about audits of British universities: “When a measure becomes a target, it ceases to be a good measure. The more a 2.1 examination performance becomes an expectation, the poorer it becomes as a discriminator of individual performances.” (A 2.1 is the second-highest class of British undergraduate degree.) Strathern credits the name “Goodhart’s law” to a 1996 essay on accountability by K. Hoskin.

The social scientist Donald Campbell reached a similar conclusion in a paper first circulated in 1976, often called Campbell’s law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

The three versions differ in emphasis. Goodhart’s is about a statistical relationship breaking down. Strathern’s is about a measure losing its value. Campbell’s adds that the activity being measured can itself be damaged.

• The record is changed, not the performance. The number is reported, classified or timed so that it looks better than what happened.
• Effort moves to what’s counted. The measured part of the job improves at the expense of the unmeasured part, as when teaching narrows to the items on one test.
• The link was never causal. A number that went along with the goal only because both had the same causes can be moved without touching the goal (see Correlation vs. causation). The reading count rose with shorter books, not with more reading.

Campbell was candid that his support was “predominantly anecdotal”. Two of his examples:

• Police clearance rates. Where police departments were judged by the proportion of crimes solved, Campbell reported (drawing on Skolnick’s research) that complaints went unrecorded, and that a burglar caught in the act could win a lighter sentence by confessing to other unsolved burglaries, in some cases ones he didn’t commit.
• Paying for test gains. In an education experiment in Texarkana, contractors paid by pupils’ score gains turned out to be teaching the answers to the specific items on the final test. Campbell concluded that achievement tests may be good indicators under normal teaching, but “when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways.”

A better-documented case comes from ambulance services in England, which were ranked partly on reaching 75% of immediately life-threatening calls within 8 minutes. Bevan and Hamblin (2009) describe official investigations that found response-time records with sharp spikes at exactly 8 minutes, implying that calls just over the line had been reassigned below it. A reanalysis found manual “corrections” in around a third of services. A later audit reported services that didn’t start the clock when the call came in, or that reclassified a call’s urgency after the fact to fit the response time achieved.

• Ask what the cheapest way to move the number would be, and whether that way also improves the thing you care about. If it doesn’t, expect some of the improvement to come from there.
• Look for signs in the data. A pile-up of results just on the right side of a threshold, like the 8-minute spike, is hard to explain by genuine improvement.
• Keep an independent check. Campbell pointed out that some statistics, like the census in his day, stayed trusted largely because nothing much depended on them. He also noted that many, himself included, assume that using several imperfect indicators will reduce the problem, and that at least one author doubted it.
• Treat the target as a question, not the answer. When a number improves after it becomes a target, ask what else changed besides the performance.

• “Tend to” is doing real work. Neither Goodhart nor Campbell claimed that every measure collapses. Campbell named indicator systems that seemed “relatively immune to distortion” and called for studying why.
• Targets can also work. Bevan and Hamblin’s main finding was that England, where the 8-minute target carried public rankings, met it while other UK countries missed it by large margins. They argued that, to have an effect, such systems need to inflict reputational damage on services that perform poorly. The same English figures contained gaming, and the authors still concluded that the target system had an effect. Goodhart’s law predicts gaming, not that a target can’t change anything.
• Not every improvement after a target is gaming. When a target or an intervention is aimed at the worst performers, some of their improvement would have happened anyway, because extreme results tend to be followed by less extreme ones (see Regression fallacy). That is a different reason a number can mislead, and it calls for a different check.
• Naming it isn’t evidence. Saying “Goodhart’s law” shows that a metric could have been corrupted, not that this one was. Like any broad principle, it explains almost anything after the fact; the question is whether there are signs of the number and the goal coming apart.