Simpson's paradox

The kidney stone treatments

A hospital compares two operations for removing kidney stones: traditional open surgery and a newer keyhole procedure through the skin (percutaneous nephrolithotomy). The keyhole procedure succeeded in 83% of cases and open surgery in 78%, so a reviewer concludes the keyhole procedure works better.

These are real figures, from a 1986 study by Charig and colleagues:

Open surgery did better for small stones and for large stones, yet worse overall. The reason is in the counts: open surgery was used mostly on large stones (263 of its 350 cases), which are harder to treat whatever the method, while the keyhole procedure was used mostly on small ones (270 of 350). Stone size affected both which operation a patient got and how likely it was to succeed, so the combined figures mostly compare hard cases with easy ones. For comparing the operations, the figures split by stone size are the fairer ones. (Even they aren’t the last word: the two operations were done in different years, with no random assignment, so other differences may remain. Charig and colleagues didn’t present the data as a paradox; statisticians later used it as an example.)

The checkout page that “loses in every group”

An online shop randomly shows half its visitors a new checkout page that nudges customers to create an account. An analyst splits the results by whether the customer created an account and finds the new page converts worse among account holders and among guests. She recommends dropping it.

Here the split is the misleading view. Whether someone created an account is partly caused by the page: the new page talked hundreds of extra, less committed shoppers into making accounts, which lowered the purchase rate among account holders while leaving only the least committed shoppers as guests. Splitting on something the change itself affects compares different kinds of people across the two pages. Because visitors were randomly assigned, the combined figures compare like with like, and they show the new page selling more: 325 purchases against 280.

The slice that tells the story

A study finds that students who use a homework app score lower on exams. A second analysis splits the students by grade and finds app users score higher in every grade. A third splits by grade and school and finds app users score lower again. Each report quotes the split that fits its author’s view.

Each additional split can reverse the pattern again, so finding a reversal proves nothing by itself. The choice of how to divide the data has to rest on what influences what (does grade level affect both app use and scores? is school a common cause, or something else?), ideally decided before seeing which way the numbers turn. Choosing the split because of the result it gives resembles drawing the target after the shots have landed, as in the Texas sharpshooter fallacy.

The Stanford Encyclopedia of Philosophy’s entry distinguishes several forms:

• Reversal: the association points one way in every subgroup and the other way, or nowhere, in the combined data, as in the examples above. This is the standard form.
• Disappearance: two things are associated in the combined data but not within any subgroup. Sleeping in your clothes goes with waking up with a headache, but among people who drank the same amount the night before, the link vanishes. This is typical of two effects of a common cause.
• Amalgamation: the combined association is stronger, or weaker, than the association in every subgroup, without necessarily changing direction.
• Trends in measurements: the same thing happens with continuous data, where a trend line through all the data slopes the opposite way from the lines within each group.

Neither view is wrong in itself. Choosing one is sound when the choice follows from how the variables affect each other:

• Split when the grouping factor influences both the treatment and the outcome. Stone size shaped both which operation patients got and whether it worked, so comparisons belong within each size. A factor like this is called a confounder.
• Combine when the grouping factor is itself affected by the treatment and you want the treatment’s overall effect, as with account creation and the checkout page. Pearl gives a medical version: a drug that works partly by lowering blood pressure should not be judged within groups defined by patients’ blood pressure after treatment.
• Trust the combined figures of a randomized comparison for the overall effect. Random assignment breaks the link between any grouping factor and who got which treatment, and without that link a systematic reversal can’t arise.
• Use combined figures to describe, as long as you don’t read them as causes. An overall admission rate is an accurate description of who got in; the error is treating it as evidence of why. In Bickel, Hammel and O’Connell’s 1975 study of graduate admissions at Berkeley, the combined figures showed “a clear but misleading pattern of bias against female applicants.” Department by department, few showed significant differences, and about as many appeared to favor women as men; women had applied more often to departments that were harder for applicants of either sex to enter. Even there, splitting by department is justified by the question being asked, which concerns the admissions committees’ decisions rather than everything that led applicants to choose where to apply.
• Sometimes neither table is right. If another, unmeasured factor also influences both treatment and outcome, splitting by the one you have may not be enough, and Pearl notes that the answer then lies in neither the combined nor the split data.

The test: is the factor you’re splitting on something that influences the treatment, something the treatment influences, or neither?

Comparing death rates by age

A retirement town has a higher death rate per 1,000 residents than a nearby college town. A health official points out that in every age group, the retirement town’s death rate is lower, and concludes that, age for age, residents there are healthier.

This is a reversal, and someone could accuse the official of choosing the flattering view. But age isn’t caused by the town; it shapes both where people live and how likely they are to die, so it is a confounder, and the official’s question, “is it healthier to live here?”, calls for comparing people of the same age. The combined rate remains an accurate description of how many deaths each town has. The choice of view follows from the causal structure, as in the split condition above.

Trusting the overall result of a randomized trial

A large trial randomly assigns patients to a new medication or a standard one and reports that the new one works better overall. A critic objects that the results weren’t broken down by sex, age or weight, and says the overall figure could be hiding Simpson’s paradox.

Subgroup results can be worth examining, since a treatment can genuinely help some groups more than others. But in a large randomized trial, sex, age and weight are spread roughly evenly across both arms, so the overall comparison isn’t a mix of different kinds of patients the way the kidney stone figures were. Random assignment is what rules out the systematic reversal, which is the randomized comparison condition above.

The combined success rate for each treatment is an average of its subgroup rates, weighted by how its cases were split. Open surgery’s 78% is weighted heavily toward its large-stone rate, because 263 of its 350 cases had large stones; the keyhole procedure’s 83% is weighted toward its small-stone rate. If the weights differ enough, the averages can come out in the opposite order from every subgroup. When the grouping factor is unrelated to which treatment was given, the weights are the same and a reversal of this kind can’t happen.

Why the reversal feels impossible is itself debated. Judea Pearl argues that people instinctively read the figures as claims about cause, and as claims about cause, a reversal really is impossible: an action that raises the chance of success in every subgroup must raise it overall, provided the action doesn’t change how many people fall into each subgroup. On this view, the surprise comes from treating an association in data as if it were the effect of an intervention. Others, including Prasanta Bandyopadhyay and colleagues, argue that the paradox is surprising even when nothing causal is involved, because people expect proportions that hold in every group to hold when the groups are combined; in their survey of university students, only 12% answered a question about this correctly. The Stanford Encyclopedia of Philosophy describes the question as a topic of continued debate.

That expectation is a close cousin of the Fallacy of composition, assuming that what is true of every part must be true of the whole. Running it the other way, assuming the combined trend must hold within each group, resembles the Fallacy of division. The practical lesson goes back to Simpson himself in 1951: depending on the story behind the data, the “sensible interpretation” sometimes lies with the combined table and sometimes with the split one.

• Ask how the groups differ in makeup. When two things are compared on a single overall figure, check whether they dealt with the same mix of cases. If not, ask for the figures within comparable groups.
• Ask whether the splitting factor came before or after the treatment. A factor fixed beforehand that influences who gets which treatment (stone size, age) is a reason to split. A factor the treatment can change (a later blood pressure reading, whether a customer made an account) is usually a reason not to.
• Sketch what causes what. Drawing arrows between the treatment, the outcome and the grouping factor is often enough to decide which view answers your question. This is the method Pearl’s analysis formalizes.
• Decide on the breakdown before looking. If a reversal only appeared after trying several ways of dividing the data, treat it as a hypothesis to test, not a finding.
• Be wary of a report that shows only one view when the groups plainly differ, whichever view it is.