Multiple comparisons problem

A community garden compares tomato yields from plots fed with 20 different homemade compost recipes against plots with plain soil. Nineteen recipes make no clear difference. Plots given the eggshell and coffee-grounds recipe produce noticeably more, a difference that is statistically significant (p < .05). The garden newsletter announces that eggshell and coffee compost boosts yields.

The 5% threshold means that a recipe that does nothing will still look significant about one time in twenty. With twenty recipes, the garden should expect about one such result even if none of them works, and the chance of getting at least one is about 64%. The eggshell recipe might really help, but this result can’t tell a recipe that works from the one that got lucky. Testing it again, on its own, could.

If every test has a 5% chance of a false positive, and the tests are independent of each other, the chance that none comes out significant is 0.95 multiplied by itself once for each test. Bland and Altman work through the case of 20 tests: 0.95²⁰ is about 0.36, so the probability of at least one significant result is 0.64: “we are more likely to get one than not.”

Tests on the same data are often related (two measures of birth weight, blood pressure every 15 minutes), and related tests inflate the chance less than this, though by an amount that’s hard to know. The expected number of false positives, 5% of the number of tests, is the same either way.

The tests don’t have to be listed in a table to count. They pile up whenever a study looks in many places:

• Subgroups. In the large ISIS-2 heart attack trial, aspirin’s overall benefit was, as Peter Sleight put it, extreme (P < 0.00001). When the patients were split by astrological sign, in Sleight’s words, “division into only 12 subgroups threw up two (Gemini and Libra) for which aspirin had a nonsignificantly adverse effect.” Bland and Altman describe a case with no real treatment at all: Lee and colleagues took records of 1,073 past coronary patients, split them at random into two “treatments” that never existed, and found a significant survival difference in one subgroup, the patients with the worst prognosis, where it would have been easy to explain.
• Many outcomes. Bland and Altman describe a pregnancy trial that reported at least 35 tests and found significant differences in two birth-weight measures that weren’t its main outcome. Applying a correction for 35 tests, neither difference is significant.
• Many subgroups crossed with many measures. In a trial of home visits for older patients, two subgroups showed significant differences (P = 0.04 and 0.03) out of at least eight subgroups on three scales. Bland and Altman calculate that even counting only the subgroups, the corrected values would be 0.32 and 0.24.
• Many measurements at once. Brain scans test many thousands of tiny cubes of the brain, called voxels, separately. Bennett and colleagues scanned a dead Atlantic salmon while “showing” it photos from a social task. Statistics without correction for multiple comparisons showed active clusters in its brain cavity and spinal column; statistics with standard corrections showed no active voxels at all, “even at relaxed statistical thresholds.” Their point was that “the probability of at least one false positive is almost certain” across the 130,000 voxels of a typical scan.
• Many analyses of the same data, when only one is reported: P-hacking.
• Many studies, when only the significant ones are published: Publication bias, the multiple comparisons problem at the scale of a whole field.

• Count the tests. A result’s strength depends on how many other chances there were. Reporting how many comparisons were made, and which were planned, lets a reader judge.
• Name the main test in advance. Sleight writes that subgroup analyses specified before a trial carry more weight than ones chosen afterward, and the same holds for a primary outcome chosen before the results are in.
• The Bonferroni correction. Divide the significance threshold by the number of tests: with five tests, require p < .01 instead of p < .05. Equivalently, multiply each p-value by the number of tests. This keeps the chance of any false positive across the whole set at about 5%. Bland and Altman note that it is too strict when tests are highly correlated, and “may miss real differences.”
• The false discovery rate. Benjamini and Hochberg proposed controlling a different quantity: the expected proportion of false positives among the results declared significant, rather than the chance of making any at all. It is less strict, so it misses fewer real effects when there are some to find. Bennett and colleagues, writing about brain imaging, call it less conservative but say it “may represent a more ideal balance between statistical power and multiple comparisons control.”
• Replication. A result found among many tests becomes evidence when it turns up again in new data, tested on its own.

Whether to correct is disputed. The arithmetic above is not controversial; the standard response to it is.

• Corrections trade one error for another. Making false positives rarer makes real effects easier to miss. Kenneth Rothman argued in 1990 that “no adjustments are needed”, that reducing false positives for null associations “increases the type II error for those associations that are not null,” and that scientists “should not be so reluctant to explore leads that may turn out to be wrong that they penalize themselves by missing possibly important findings.”
• A result’s meaning shouldn’t depend on unrelated tests. Thomas Perneger argued that the Bonferroni method answers a question researchers rarely ask, whether all the null hypotheses are true at once, and that under it “the interpretation of a finding depends on the number of other tests performed.” He concluded that “simply describing what tests of significance have been performed, and why, is generally the best way of dealing with multiple comparisons.” He also notes situations where adjustment does make sense, as when the question really is whether any of many comparisons shows a difference.
• The problem may be the model, not the number of tests. Andrew Gelman, Jennifer Hill and Masanao Yajima argue that “the problem of multiple comparisons can disappear entirely when viewed from a hierarchical Bayesian perspective”: instead of widening every interval, a model that treats the comparisons as related pulls the estimates toward each other, which makes the comparisons “appropriately more conservative.”

What the positions share is that a reader needs to know how many tests were run and which were planned. Every correction depends on that count, and so does Perneger’s alternative of describing what was done.

A corrected “not significant” isn’t “no effect.” A result that fails after correction is uncertain, not disproved. The eggshell compost may still work; the test just can’t show it yet.

A lucky result can still be true. The problem is about how much a result can prove, not about whether it’s right. Some of the significant results found among many tests are real. What the principle says is that you can’t tell which from the result alone.