Survivorship bias

“They built things to last back then”

Walking through an old neighborhood, someone says: “Look at these houses from 1910, solid brick and still standing after more than a century. Nothing built today will last like that.”

The clear-cut case. Houses from 1910 that were flimsy, badly built or badly placed have mostly been torn down, burned or replaced, so the ones left to look at were filtered on exactly the quality being judged. A fair comparison would need all the houses built in 1910, including the ones that are gone. Older building may or may not have been better; the houses that are still standing can’t settle it.

The funds on the list

An investment newsletter collects the ten-year returns of every stock fund a brokerage currently offers and reports that the average fund beat the market. “Professional managers earn their fees.”

“Currently offered” is a filter. Funds are closed or merged away, and Edwin Elton, Martin Gruber and Christopher Blake note that funds that disappear tend to do so because of poor performance. Their returns drop out of the ten-year average, which then describes the funds that lasted. To measure the size of this bias, Elton and colleagues tracked every fund that existed at the end of 1976, including the returns of funds that later merged. Stephen Brown and colleagues showed a subtler effect: in a sample truncated by survivorship, the relationship between volatility and returns can make past performance appear to predict future performance.

Hits on the planes that came back

During the Second World War, analysts tally where bombers returning from missions were hit. The engines carry a smaller share of the hits than their share of the plane’s surface. Someone reading the tally concludes that the engines are rarely hit and need the least protection.

The tally comes only from planes that returned. In a 1943 memorandum, the statistician Abraham Wald pointed out that if all the hits on returning planes were in one area, there would be two possible explanations: hits never land anywhere else, or a hit anywhere else always downs the plane. The survivors’ damage alone can’t tell these apart. His method brought in information from outside the filter, how hits would be spread over the plane if none were lost (estimated, for example, by firing dummy bullets), and compared it with the survivors’ damage. A part with fewer hits than expected among the survivors is a part where hits bring planes down. In Wald’s own worked example, which used hypothetical data, the engines came out as the most vulnerable part.

Survivorship bias is a form of Selection bias: Alexandra Ortego and colleagues describe it as “a specific type of selection bias that occurs when analysis is restricted to individuals or entities that have survived a particular disease or condition”, and the first causal diagram of selection bias in Hernán and Robins’s textbook is a study restricted to fetuses who survived until birth.

• Survivors of a disease. A study that enrolls people who have lived with a condition for some time misses those who died quickly and those who recovered. The Catalogue of Bias calls this prevalence-incidence (Neyman) bias: excluding patients who have died makes a disease look less severe, and excluding those who recovered makes it look more severe.
• Survivors of time. Old buildings, long-lived companies, books still in print: whatever is left from the past is a filtered sample of what existed.
• Success stories. Advice drawn from the people who succeeded, without the many who did the same things and failed. Testimonials and profiles of the rich work this way; see Anecdotal evidence and Appeal to wealth.
• A two-factor filter. When survival depends on either of two qualities, survivors show a spurious trade-off between them: a restaurant that lasts in a bad location probably has good food. This structure is Collider bias.

A related problem: immortal time bias. In some cohort studies, people count as “treated” only once they fill a prescription, so they must have survived until then. The Catalogue of Bias describes how this hands the treated group a period in which, by design, they couldn’t have had the outcome, making treatments look more protective than they are.

Looking at survivors is legitimate when:

• The conclusion is about the survivors themselves. What long-term survivors of an operation need from a clinic is a question about survivors.
• The failures are in the data. A fund study that keeps the full record of every fund that existed at the start, including the ones that closed, can compare survivors with everyone.
• The filter is unrelated to the question. If the houses that remain were chosen for demolition at random, their sturdiness would tell you about the rest.
• You know what the unfiltered picture would look like. Wald could interpret the survivors’ damage because he had an independent estimate of where hits would land.

The test: what would the cases that didn’t make it look like, and could they change the answer?

Planning a clinic for survivors

A hospital surveys patients who are alive five years after a particular heart operation to learn which rehabilitation services they still use, and plans its outpatient clinic around the answers.

The sample is survivors only, but the clinic will serve survivors only, so the filter matches the question. It would become survivorship bias if the same survey were used to judge how well the operation works, since those who died aren’t in it. That’s the about the survivors themselves condition.

Studying funds that include the dead ones

A researcher uses a database that keeps the full return history of every fund that existed ten years ago, including those later closed or merged. She reports the average for all of them, and separately notes that the funds still open did better than that average.

She does look at survivors, but alongside the failures, so readers can see what the filter did. Her statement about the funds still open is accurate, and her overall figure isn’t inflated by missing funds. That’s the failures are in the data condition.

Failures tend to leave no record where we look. Closed funds disappear from the list, demolished houses from the street, lost planes from the tally, and people who tried and failed rarely write books about it. The cases in front of us feel like the whole picture.

The filter is also easy to mistake for a neutral description. “Funds currently offered” or “planes that returned from missions” sound like a way of naming the data, not a way of selecting them. Once the filter is out of sight, whatever helped cases survive looks like a general feature of the kind.

• Name the filter. Ask what a case had to do to end up in front of you: stay open, stay standing, come back, get noticed.
• Ask where the failures went, and whether any record of them exists. Fund databases that retain dead funds exist for this reason.
• Start from the beginning, not the end. Compare everything that started (all funds open ten years ago, all houses built in 1910) rather than what remains.
• Ask what you’d expect to see with no filter. Wald’s move: the survivors’ pattern means something only against an estimate of the pattern before anything was lost.
• Don’t swing to the opposite. The survivors’ features may really have helped. The bias means the survivors alone can’t show it.

Survivorship bias is a flaw in method, not an effect with a replication record. Its two best-known settings have been studied directly.

• Wald’s aircraft work. Abraham Wald’s analysis, written in 1943 for the Statistical Research Group at Columbia University, consists of eight memoranda totaling over 100 pages. The Center for Naval Analyses reprinted them in 1980, and Marc Mangel and Francisco Samaniego published an exposition in
  1984. It is mathematical: equations relating the observable proportions of returning planes with a given number of hits to the unknown probabilities of surviving each hit, bounds and approximations, confidence limits, and an extension to different parts of the plane and different guns. Wald was explicit that the hit distribution and each part’s vulnerability can’t both be estimated from survivors alone. His worked examples use hypothetical data, and he wrote that conclusions of this kind could be used as guides for locating protective armor. Mangel and Samaniego report that his methods were used in the Second World War and by the Navy and Air Force in the wars in Korea and Vietnam.
• The popular story. The popular version adds a scene in which military officers propose armoring the most-hit areas and Wald overrules them. Neither the reprinted memoranda nor Mangel and Samaniego’s account describe such an exchange. The documented lesson is the one in the memoranda: damage to survivors can be interpreted only with an independent idea of what the damage would look like if no planes had been lost.
• Fund performance. Brown, Goetzmann, Ibbotson and Ross (1992) showed with numerical examples that truncation by survivorship can be strong enough to account for the evidence that past mutual fund performance predicts future performance. Elton, Gruber and Blake (1996) estimated survivorship bias over different horizons and performance models by following every fund that existed at the end of 1976, including funds that merged.