Attrition bias

Happier with every survey

A research team surveys 2,000 adults aged 70 every two years about their life satisfaction. By the fifth survey, 1,100 are still answering, and their average satisfaction is higher than the whole group’s was at the start. A newsletter reports: “Study finds people grow more satisfied as they age.”

The clear-cut case. People stop answering a long survey for many reasons, and some of the likeliest are poor health, bereavement or moving into care, all of which also lower life satisfaction. The average has risen partly because the less satisfied participants left the pool, not necessarily because anyone’s satisfaction rose. Comparing each person with their own earlier answers, and checking whether those who later dropped out started out less satisfied, would show how much of the rise is real.

“Dropout was balanced”

A trial compares a new stretching routine with a standard exercise sheet for back pain. After three months, 30% of each group has stopped coming to assessments. The report notes that dropout was balanced between the groups, analyzes the people who remained, and finds less pain in the stretching group.

The less obvious case, because equal rates look reassuring. Suppose people on the exercise sheet mostly left because it wasn’t helping and they went elsewhere for treatment, while people doing the stretches mostly left because their backs felt better and the appointments no longer seemed worth it. Then the remaining exercise-sheet group has had its successes kept and its failures removed, and the stretching group the reverse. The same percentage leaving each group says nothing about whether the same kinds of people left. Bell and colleagues call “equal dropout means no bias” a myth (see Evidence).

Analyzing the people who stuck with it

A trial randomly assigns office workers to a lunchtime walking group or no program. Only people who attended at least 80% of the walks are included in the analysis, “so that we measure the effect of actually doing the program”. The walkers who were included have lower blood pressure than the comparison group.

Here nobody vanished; the investigators removed people themselves. Workers who managed to attend nearly every walk are likely to differ from those who didn’t: less busy, healthier to begin with, more motivated. Comparing them with an entire unselected group mixes the effect of walking with the effect of being the kind of person who shows up. Schulz and Grimes put the rule for the main analysis bluntly: exclusions after randomization, in principle, “none are allowed.”

• Loss to follow-up. Participants can’t be reached for the final measurement, as in the survey example.
• Withdrawal and dropout. Participants choose to leave, often because of side effects, lack of benefit or recovery, as in the back-pain trial.
• Exclusions after randomization. Investigators leave some assigned participants out of the analysis, for instance those who didn’t follow the protocol (sometimes reported as a per-protocol analysis), as in the walking example.
• Missing outcome data within a study. People remain enrolled but skip the questionnaire or test on the day it counts. The Cochrane Collaboration’s 2011 risk of bias tool assessed attrition bias under the item “incomplete outcome data”.

Where it’s filed. Hernán, Hernández-Díaz and Robins give informative censoring in cohort studies (losing people for reasons tied to their outcome) as an example of Selection bias: the analysis is restricted to people whose staying in the study was influenced both by the exposure (or its causes) and by the outcome (or its causes). The Cochrane Collaboration’s 2011 tool instead lists attrition bias as its own domain, separate from what it calls selection bias (flaws in how participants are allocated to groups). Attrition bias means the same thing in both; what differs is how much the label “selection bias” covers.

Not the same error: survivorship bias. In Survivorship bias, the failures were gone before anyone started looking: the funds that closed, the planes that didn’t return. In attrition bias, the study enrolled the people who later left, so their starting data and often the reasons they left are on record. That makes it possible to check how the leavers differed and to estimate what their absence did to the result.

Losing participants doesn’t automatically bias a study. Results can be trusted when:

• People left for reasons unrelated to the treatment and the outcome. The Catalogue of Bias notes that such losses may have little or no impact on the results. Someone who moves abroad for work is unlikely to differ in a relevant way from someone who stays.
• Everyone was analyzed as randomized. In an intention-to-treat analysis, all participants are analyzed in the group they were assigned to, whether or not they followed the program. If outcomes were collected even from those who stopped the treatment, stopping didn’t remove them from the data.
• The reasons for leaving are captured by what was measured, and the analysis uses that. Bell and colleagues show that when dropout depends only on information recorded before people left (for example, their earlier scores), methods such as likelihood-based mixed models give unbiased estimates even when dropout differs between groups. That assumption can’t be proven from the data, so it should be paired with a sensitivity analysis.
• The conclusion survives a worst case. The Catalogue of Bias suggests assuming a worst-case outcome for everyone with missing data; if the study’s conclusions don’t change, the losses are likely not a threat to its validity.

The test: could the reasons people left be related to how they were doing, and would the result still hold if they had done worse than those who stayed?

Unequal dropout, handled

A trial of two programs for building reading fluency loses 25% of one group and 10% of the other. A reviewer calls the result biased because dropout was so uneven. The authors’ report shows that children who left had lower scores at their last assessment, analyzes all children with a mixed model that uses those earlier scores, and adds a sensitivity analysis in which every missing child is assumed to have made no progress. The difference between programs holds up in both.

The uneven rates are a reason to look, and the authors looked. Dropout was tied to something they had measured, their analysis used every child’s recorded scores rather than only the finishers’, and a pessimistic assumption about the missing children didn’t overturn the result. That’s the reasons captured and used and survives a worst case conditions. Bell and colleagues describe the belief that unequal dropout necessarily biases a result as a second myth.

A handful who moved away

A two-year study of a school gardening program follows 600 pupils. Eighteen are lost, all recorded as having changed schools when their families moved, in similar numbers from the gardening and comparison schools. A parent says the results can’t be trusted because not everyone was followed up.

Eighteen of 600 is 3%, the reasons were recorded, and a family relocation has no obvious link to what the program could change. The parent is right that not everyone was measured, but nothing suggests the missing pupils were doing better or worse than the rest. That’s the unrelated reasons condition.

The people who stay are the ones with data, so they are the ones who get analyzed. Analyzing only the completers also feels methodologically cleaner: they received the full treatment, so their results seem to show what it “really” does. That intuition is exactly what makes the bias hard to see, and Schulz and Grimes warn that some explanations for mishandling exclusions “intuitively appeal to readers, disguising the seriousness of the issues.”

The reasons people leave are also tangled up with the thing being studied. Side effects, disappointment, recovery, illness and changes in circumstance all make people less likely to keep showing up, and all of them bear on the outcome. And summary checks give false comfort: a balanced dropout rate or a randomized design looks like protection, but randomization, as Schulz and Grimes put it, “means little if investigators cannot include all randomised participants in the primary analysis.”

• Find the numbers. How many started in each group, how many were assessed at the end, and why the others weren’t? Schulz and Grimes recommend that reports show the progress of every randomized participant, for instance with a trial profile. The survey example is a reminder to ask the same thing of any study that follows people over time.
• Compare leavers with stayers. Did the people who dropped out already differ at the start, or were they doing worse at their last measurement?
• Ask whether everyone was analyzed as assigned. A result based only on completers, or only on participants who followed the protocol, is a comparison of selected groups.
• Look for a sensitivity analysis. Would the conclusion change under a best-case and worst-case assumption about the missing outcomes?
• Don’t judge by the dropout rate alone. Balanced rates don’t show there is no bias, and some attrition doesn’t show there is. The Catalogue of Bias cites a rule of thumb that under 5% attrition leads to little bias and over 20% poses serious threats to validity, but notes that even small losses can cause significant bias.

Attrition bias is a flaw in method rather than an effect with a replication record, but studies have estimated how much loss to follow-up could change published results, and when it does.

• Akl and colleagues (2012) reviewed 235 reports of randomized trials in five leading general medical journals (2005–07) that reported a significant result on a binary outcome important to patients. 13% didn’t say whether anyone was lost to follow-up. Among those that did, the median loss was 6% (interquartile range 2–14%). Under assumptions the authors considered more plausible, in which the event rate among people lost to follow-up, relative to those followed up, was higher in the treatment group than in the control group, the results of 0% to 33% of trials were no longer significant. Under a worst-case assumption, 58% were not.
• Bell and colleagues (2013) used data from a trial in advanced kidney cancer, in which 64% of the control group and 70% of the experimental group dropped out of the quality-of-life measurements, plus a simulation. Starting from a reconstructed complete dataset and deleting observations in a way that depended on earlier recorded scores, they showed that an analysis of complete cases could be badly biased with near-equal dropout (an estimate 38% too low) and more so with unequal dropout (72% too low, or 74% too high), while a mixed model was unbiased or much less biased. Their conclusion was that whether dropout differs between groups “can be a red herring”; what matters is why data are missing and how they are analyzed.
• Hewitt and colleagues (2010) examined individual patient data from 10 trials of treatments for musculoskeletal disorders, with attrition from 4% to 28%. In their words, “there was no indication that attrition altered the results in favor of either the treatment or the control” in the individual trials. They caution that the sample was small, and only 2 of the 10 trials had attrition above 15%.

What remains uncertain is how often attrition changes conclusions in practice, as opposed to how much it could. The answer depends on why people left, which usually can’t be observed directly, so estimates rest on assumptions about the missing data.