Selection bias

The course that works for everyone who finishes

An online coding bootcamp runs for 12 weeks. Of 500 people who start, 180 finish, and the finishers’ scores on a programming test rise by an average of 40 points. The advertisement says: “Our bootcamp raises test scores by 40 points.”

The clear-cut case. The 320 who left aren’t a random selection of the starters: people who fell behind or found the material too hard are the likeliest to quit. The 40 points describes the people who kept going, who were probably learning fastest. The Catalogue of Bias calls this attrition bias: people who leave a study differ systematically from those who continue. (The advertisement has a second problem, no comparison with people who didn’t take the course, but even setting that aside, the figure can’t be applied to someone signing up.)

Counting from the repair log

A bike shop wants to know how often riders in town get flat tires. It goes through a year of repair records: customers who brought a bike in had an average of 2.5 flats. The owner tells the local cycling club that “the typical rider here gets two or three flats a year.”

Nobody volunteered and nobody dropped out, yet the sample is selected. A rider appears in the log only by bringing a bike in, riders with more flats come in more often, and riders who never had a flat or who patch their own tubes never appear at all. The records are an accurate account of the shop’s repair customers and a poor guide to riders in general. Epidemiologists meet the same problem when they count a disease from medical records, which miss people who never sought care (see Evidence).

Dropouts in a randomized trial

A trial randomly assigns 300 people with knee pain to a daily walking program or to usual care. After six months, 90% of the usual-care group come back for the final assessment, but only 60% of the walking group, many of whom stopped because walking made their knees hurt. Among those assessed, the walking group reports less pain, and the authors conclude that the program reduces pain.

The less obvious case, because randomization looks like a guarantee. It made the two groups comparable at the start, but the comparison is made only among people who stayed, and staying depended both on the program and on how bad each person’s knees were, which also affects pain six months later. The walkers who remained are disproportionately those for whom walking didn’t hurt, a group already headed for less pain. Hernán and Robins put it directly: randomization protects against confounding, but not against selection bias when the selection happens after randomization.

“Selection bias” is used for two related problems, and it helps to keep them apart:

The Catalogue of Bias’s definition covers both: individuals or groups in a study differ systematically from the population of interest, leading to a systematic error in an association or outcome. Hernán, Hernández-Díaz and Robins (2004) reserve the term for the second. On their structural definition, selection bias is bias from conditioning on a common effect of two variables, one of which is the exposure or a cause of it and the other the outcome or a cause of it; bias from common causes is Confounding. Their book notes that statisticians and econometricians often use “selection bias” for both kinds, and Neil Pearce and Lorenzo Richiardi describe the traditional meaning as bias from inappropriate selection, or self-selection, of study subjects from the source population.

• Self-selection and volunteer bias. People who choose to take part differ from those who don’t. The Catalogue of Bias notes that volunteers tend to be more educated and of higher social class, and that the bias can operate at every stage, from recruitment to follow-up.
• Attrition bias (loss to follow-up). People who leave a study differ from those who stay, as in the bootcamp and the knee trial. Hernán and Robins also call it bias due to informative censoring.
• Non-response bias. People who don’t answer a survey differ from those who do. The Catalogue of Bias notes that people in poorer health tend to avoid health surveys, and those who take part report better health.
• Healthy participant and healthy worker effects. People well enough to join a long-term study, or to stay in a job, differ from the general population. As Confounding notes, “healthy worker bias” is used for two different structures, one a selection problem and one a confounding problem.
• Survivorship bias. Only the cases that made it through some filter are studied: the funds still open, the planes that came back. See Survivorship bias.
• Berkson’s bias and other collider biases. Selection that depends on two conditions creates an association between them among the selected, as in hospital data. See Collider bias, which also covers the dispute over whether selection bias is a kind of collider bias or the other way around.

A self-selected or filtered sample is also what a Hasty generalization often rests on: the generalization is the flawed argument, and selection bias is the flaw in the data behind it. The same logic applied to which studies get published, rather than which people get studied, is Publication bias.

A selected sample doesn’t automatically mean a biased conclusion. It’s sound when:

• The sample was drawn at random from the population the conclusion is about. Chance still makes estimates imprecise, but not systematically tilted.
• The conclusion is limited to the group that was selected. The repair log is good evidence about the shop’s repair customers.
• Selection is unrelated to what’s being studied. The Catalogue of Bias notes that if people leave a study for reasons unrelated to the exposure and the outcome, it may have little or no effect on the results.
• Selection happened before random assignment. Hernán and Robins point out that randomized trials enroll only volunteers but aren’t affected by volunteer bias in their comparison, because treatment is assigned after people agree to take part. Whether the result applies to other kinds of people is a separate question.
• The selection has been accounted for. Weighting based on the factors that drove selection can correct for it under certain assumptions, and a worst-case analysis of the missing cases may show the conclusion survives.

The test: could the way cases got into, or out of, the data be related to what’s being measured?

A trial of volunteers

A trial recruits 400 volunteers with seasonal allergies, randomly assigns half to a new nasal spray and half to a placebo spray, and follows nearly all of them to the end. The spray group has fewer days with symptoms. A commenter objects that volunteers aren’t typical allergy sufferers, so the result must be selection-biased.

Volunteers may well differ from other allergy sufferers. But everyone was a volunteer before the coin was flipped, and almost no one dropped out, so the two arms are comparable and the difference between them is a fair estimate of the spray’s effect in people like these. The commenter’s real point is about generalizing the result, which is worth raising on its own terms. That’s the selection before random assignment condition.

Staffing the busiest lifts

A ski resort’s gates scan every pass each time a skier boards a lift. Over the season, the resort finds which lifts are busiest at which hours and moves staff to them.

The data cover only people who ski at this resort, not skiers in general. But the decision is about this resort’s lifts and this resort’s skiers, and the gates record every ride, not a sample of volunteers or responders. The selection matches the conclusion: the limited to the group that was selected condition.

The cases in front of you feel like the whole story, a pattern Daniel Kahneman calls “what you see is all there is”. The people who quit, never answered or never came into the shop leave no trace in the data, so nothing in the numbers prompts the question of who is missing. The selection is a fact about the process that produced the data, not something visible in the data themselves.

Size adds false reassurance. A sample of thousands feels trustworthy, but selection bias is systematic, so it doesn’t shrink as the sample grows. And in the structural sense, no one needs to choose anyone: participants’ own decisions to stay or leave are enough, which is why it can undo the protection a randomized design seems to offer.

• Ask how cases got into the data, and who couldn’t have. The Catalogue of Bias recommends reporting how many people were screened, how many were included, and how many were lost along the way.
• Compare those included with those left out on whatever is known about both. If the finishers already differed from the dropouts at the start, their results can’t stand for everyone.
• In trials, ask whether everyone was analyzed as randomized. An intention-to-treat analysis includes all participants in the groups they were assigned to, and best-case and worst-case scenarios for the missing outcomes show how much the losses could matter.
• Shrink the claim to the sample. “Among our repair customers” is often the honest version.
• Don’t use “selection bias” as a blanket dismissal. The Catalogue of Bias notes that its size and even its direction are often hard to determine. The objection has force when it names a plausible route by which selection is tied to the result.

Selection bias is a flaw in method rather than an effect with a replication record, but its size has been estimated where the missing cases could be found. The examples below are as summarized by the Catalogue of Bias.

• Cases that records miss. A door-to-door survey of Parkinson’s disease in a US county reached over 97% of households and found 31 cases among about 24,000 residents. Thirteen had never been seen for medical care, so a count based on the medical care system would have missed over 40% of them.
• Survival estimates. A systematic review of cohort studies of preterm infants found that selection bias overestimated survival by as much as 100%.
• Loss to follow-up in trials. A review of 160 randomized trials in five leading medical journals, with an average loss to follow-up of 6%, found that, depending on the assumptions made about the outcomes of those lost, between 0% and 33% of the trials would no longer have statistically significant results.
• Healthy participants. Prospective cohort studies of diet and lifestyle report lower death rates among participants than in the general population, suggesting that people with healthier habits are more likely to sign up.