Anecdotal evidence

My sister’s car

A consumer survey of 40,000 owners rates a car model well below average for reliability. “That’s nonsense. My sister has driven one for eleven years and it’s never needed anything but oil changes.”

The clear-cut case. The sister’s experience is real, and it’s the kind of thing a below-average rating allows: plenty of owners of an unreliable model still have no trouble. The survey already includes thousands of cars like hers, along with the ones that broke down. One good car can’t outweigh the count, because the count is exactly what a single car can’t show.

The tonic that cured a cold

“I felt a cold coming on, so I drank my aunt’s ginger-and-honey tonic three times a day. Two days later I was fine. That stuff really works.”

Here the story supports a claim about cause, not frequency. Colds usually get better on their own within days, so the speaker would likely have recovered with or without the tonic. A single experience has no comparison: nobody knows what would have happened that week without it. Patrick Hurley and Lori Watson make the same point about a story of a remedy followed by recovery: it ignores the people who took the remedy and didn’t recover, and the people who recovered without it. Mistaking “after” for “because of” is Post hoc ergo propter hoc.

Twenty-five years of experience

A district trials a new reading program in 30 schools and finds modest gains overall. A teacher with 25 years in the classroom tells a meeting: “The one year my school used it, I had the worst reading scores of my career. I’ve seen what it does. It doesn’t work.”

This sounds like the voice of experience, and the teacher is well placed to notice real problems with how a program runs. But her evidence on whether it raises scores is one class in one year, and classes differ a great deal from year to year for reasons that have nothing to do with the program. The trial compared many classes, which is what lets those differences average out. Expertise makes a story more credible as a description; it doesn’t turn one case into a sample.

• A story against the statistics: an experience offered to override systematic evidence, as with the car. Bradley Dowden’s list of fallacies in the Internet Encyclopedia of Philosophy describes the error as discounting evidence from systematic search or testing in favor of a few firsthand stories.
• A story about cause: “I tried it and it worked”, where the missing comparison is what would have happened otherwise.
• Collections of stories: testimonials, reviews and success stories. Many anecdotes still aren’t a sample if they were chosen for being positive, or if only people with good results tell them; the silence of the others is Survivorship bias.
• Misleading vividness: the name Dowden’s list gives to jumping to a conclusion because of the special emphasis placed on an anecdote or another vivid piece of evidence. A long, moving story can crowd out a dry number even when the number rests on far more cases.
• “In my experience”: an expert’s memorable cases, as in the reading-program example. Real expertise, but still a small and unsystematic sample for questions of frequency or effect.

Dowden classifies anecdotal evidence as fallacious generalization from an inadequate sample, which makes it a form of Hasty generalization: a generalization from a sample of one. Hurley and Watson discuss it in their chapter on science, where anecdotal evidence is ruled out because it is too isolated to establish a causal connection.

A story is evidence, just limited evidence. Using one is sound when:

• The claim is only that something can happen. One real case proves possibility: “people can get a refund after the deadline; my neighbor did.”
• The claim being tested is universal. A single counterexample defeats “always” or “never”, however large the evidence for the general tendency.
• It’s treated as a lead, not a conclusion. In medicine, a well-documented case report of an unexpected effect is often the first sign of something new. Jan Vandenbroucke describes case reports as highly sensitive for detecting novelty, a source of many new ideas in medicine and one of the cornerstones of medical progress.
• It illustrates what systematic evidence already shows. A story that makes a well-established statistic concrete isn’t standing in for the evidence; it’s explaining it.
• Nothing better is available and little rides on it. Picking a restaurant on a friend’s recommendation is reasonable, provided you’d revise the choice if better information turned up.

The test: is the story being used to show that something can happen, or to show how often it happens or what causes it?

One wet sock

A hiking boot is advertised as “completely waterproof.” A buyer stands in ankle-deep water for five minutes, well below the top of the boot, and water seeps in at the toe. “So much for completely waterproof.”

One pair of boots, one test, and the buyer rejects a claim about the whole product line. That’s legitimate, because “completely waterproof” is a universal claim, and one counterexample defeats a universal claim. The buyer would be making the error if they concluded that the brand’s boots usually leak.

A case report

A doctor sees a patient develop a rare, specific reaction soon after starting a newly approved medication, rules out the usual causes, and writes it up as a case report, suggesting that other doctors watch for the same reaction.

One patient, and a possible causal link. But the doctor doesn’t conclude that the drug causes the reaction, or how often. The report is offered as a signal for further study, the kind that case reports are good at. That’s the lead, not a conclusion condition: the claim is sized to a single case.

Stories are concrete and easy to remember; statistics are abstract. When people judge how common or likely something is by how easily examples come to mind, the Availability heuristic, one vivid case can outweigh a large number that has no faces attached. A personal experience also feels like direct observation, while a survey is secondhand. And a story never shows what’s missing from it (the cars that broke down, the colds that cleared up without tonic), so it seems complete. That’s Daniel Kahneman’s “what you see is all there is”.

The pull is strongest when a lot feels at stake. Stories tend to win when the issue is threatening, about health, or personal, and statistics tend to win when it isn’t (see Evidence).

• Ask what the story can show. Usually the answer is “that it’s possible”, which may not be in dispute.
• Ask about the comparison. What happened to people in the same situation who didn’t take the remedy, buy the product or follow the advice?
• Ask whether the systematic evidence already includes cases like this one. It usually does.
• Don’t dismiss the experience. It happened; the question is only what it can support. Saying “I believe you, and it’s still one case” keeps the conversation about the evidence.
• Make the numbers concrete. In two survey studies with hypothetical treatment choices, Angela Fagerlin and colleagues (2005) found that showing cure rates as a pictograph (a grid of figures) reduced the influence of patient stories on people’s choices, while a quiz about the tradeoffs did not.

Anecdotal evidence is a matter of reasoning rather than an empirical effect, but researchers have measured how much stories sway judgments.

• Freling and colleagues (2020) meta-analyzed 61 papers on whether anecdotal or statistical evidence persuades more. The mixed earlier findings lined up with emotional engagement: anecdotes were more persuasive when engagement was high (a severe threat, a health issue, or something affecting the reader personally), and statistics were more persuasive when it was low.
• Fagerlin, Wang and Ubel (2005) asked people at a courthouse and an airport to choose between two treatments for a heart condition. When cure rates were given in prose, the choice tracked patient stories: 41% chose bypass surgery when the stories matched the statistics, and 20% when the stories were unrepresentative of them. A pictograph of the cure rates reduced the stories’ influence; a quiz did not. The scenarios were hypothetical, so the study shows how people weigh the information, not what they would do as patients.