Suppressed evidence

The trial that went great

A team lead makes the case for switching every team to a new project-management tool: “Our trial went really well. The design team loves it, their tasks close faster, and onboarding took an afternoon.” The trial report she has read also shows that two of the five teams in the trial went back to the old tool within a month.

The clear-cut case. Everything she says is true, but the evidence she leaves out bears directly on her conclusion, which is about every team. A trial where three teams stayed and two left is a very different result from the one she describes. She hasn’t said anything false; the argument misleads by what it omits.

Choosing where the chart starts

“Since the website redesign, newsletter sign-ups have tripled,” says a slide, showing sign-ups from January, the month of the redesign, to June. Last year’s figures show sign-ups always drop sharply in December and January and recover by spring.

No data point is hidden inside the chosen months. The selection is in the starting point: beginning at the yearly low makes any spring look like a surge. Setting this year’s rise beside last year’s January-to-June rise, or comparing this June with last June, would show how much of the “tripling” is just the usual seasonal pattern.

Four studies that agree

A student writing a paper on whether background music helps people concentrate searches for “music improves concentration,” reads the first several studies that come up, all of which found a benefit, and writes: “Research shows that background music improves concentration.”

Nobody here is hiding anything, and the studies are real. But the search phrase asked for studies that found a benefit, so that’s what turned up. A search for studies that found no effect, or a harmful one, would likely have turned some up too. The conclusion speaks for “research” when it rests on the part of the research that happened to agree. This is cherry-picking without intent, and it’s how confirmation bias usually works in practice. Bradley Dowden’s list of fallacies notes that evidence overlooked by accident counts as suppressed evidence too, even though the name suggests a deliberate act.

Rudolf Carnap called the principle being broken the requirement of total evidence: when judging how probable a conclusion is, the total evidence available must be taken as the basis. He noted that it has no counterpart in deductive logic, where a valid argument stays valid whatever else you learn, and that its validity can’t be doubted: a judge who ignored relevant facts, or a scientist who left unfavorable experimental results out of a publication, would be regarded by everyone as doing something wrong.

Hurley and Watson classify suppressed evidence as a fallacy of presumption: the argument works by creating the presumption that its premises are both true and complete when they are not. They define it narrowly, as ignoring evidence that outweighs the evidence presented and would lead to a different conclusion.

• Leaving out known counterevidence: the classic case, as in the project-management trial.
• Choosing the comparison: a starting date, a baseline, or a raw count where a rate is needed, as in the sign-ups chart. Hurley and Watson describe statistical varieties of suppressed evidence, such as comparing counts across a period in which the population they’re drawn from also grew.
• Quoting out of context: keeping the words that serve the argument and dropping the ones that change their meaning. Hurley and Watson list this as a form of suppressed evidence.
• Selective citation: citing the studies that agree and not the ones that don’t. In science the selection can also happen before any arguer sees the evidence: studies with null results go unpublished (Publication bias), or only the analyses that came out significant get reported (P-hacking). Someone summarizing that literature in good faith can pass the selection along.
• Anecdotes as the chosen evidence: when the evidence picked is a single story set against the rest, see Anecdotal evidence.

Not the same error: the Texas sharpshooter fallacy. In cherry-picking, the conclusion comes first and evidence is chosen to fit it. In the Texas sharpshooter fallacy, the data come first and the conclusion (the “target”) is drawn around whatever pattern they happen to show. Both leave out the data that don’t fit, and they often occur together.

Not every textbook treats suppressed evidence as a fallacy in its own right. Douglas Walton, in a book-length study of bias in argument, notes that some treat it instead as a deficiency in an argument, and argues that how much one-sidedness is acceptable depends on the kind of exchange: a sales pitch is not held to the standard of a critical discussion.

Every argument leaves things out. Selecting evidence is fine when:

• The selection is labeled and claims no more than it shows. “Here are three examples of” or “our best results” tells the audience they’re seeing a selection, and doesn’t pretend it’s typical.
• The summary rests on all the evidence. A systematic review, which searches for every relevant study by a method set in advance and reports how it chose them, can then be summarized in a sentence without suppressing anything.
• What’s left out wouldn’t change the conclusion. Omitting weak, irrelevant or redundant evidence is ordinary brevity. The fallacy is leaving out evidence that would shift the balance.
• The other side has its own voice. In a debate, a trial with opposing counsel, or a meeting where each option has an advocate, a one-sided case is expected, because the missing evidence will be put forward by someone else. The error is taking one side’s case as the whole picture.

The test: would the evidence left out change the conclusion, and does the audience know they’re seeing a selection?

A portfolio

A designer applying for a job sends a portfolio of her ten best projects from the past five years.

She has chosen only her best work and left out everything else. But everyone involved knows what a portfolio is: a selection meant to show what she can do at her best, not an average of her output. The employer can ask for references or a trial project for the rest. That’s the labeled selection condition.

One sentence from a review

A gardening leaflet says: “A review of 25 field trials found that mulching raised tomato yields in most of them,” and doesn’t describe any of the individual trials.

The leaflet mentions none of the trials where mulching made no difference, which can look like cherry-picking. But the sentence reports a finding drawn from all 25 trials, including the unfavorable ones, and says “most”, not “all”. The leaving-out happened in the summarizing, not in the choosing of evidence. That’s the summary rests on all the evidence condition. If the review itself had searched only for trials that worked, the problem would be back.

Deliberate cherry-picking is easy to explain: an advocate wants to win. The inadvertent kind is more common and harder to see. People look for information in ways that favor what they already believe, a pattern studied as confirmation bias. In a perceptual task, Paula Kaanders and colleagues (2022) found that after making a choice, people sampled more information about the option they had chosen, and more so the more confident they were; as a result, they were more likely to stick with the original choice even when it was wrong. When the experimenter controlled what information people saw, the effect disappeared.

It also works on audiences because missing evidence leaves no trace. A list of true, favorable facts looks complete unless you already know what should be there. That’s Daniel Kahneman’s “what you see is all there is”.

• Ask what the whole set looks like. All the teams in the trial, every month of the year, every study on the question, not just the ones mentioned.
• Ask how the evidence was found. A search phrased to find support will find it; a comparison starting at a low point will show a rise.
• Look for a summary of all the evidence, such as a systematic review, before trusting a handful of studies.
• In your own arguments, go looking for the strongest evidence against your conclusion and say what it is. If the conclusion survives, the argument is stronger for it.
• Don’t conclude the opposite. Showing that evidence was cherry-picked shows the case is incomplete, not that its conclusion is false. The full picture may still support it.

Suppressed evidence is a flaw in reasoning rather than an effect with a replication record, but its research form, selective citation, has been measured.

• Greenberg (2009) mapped every English-language paper indexed in PubMed that addressed one claim in muscle-disease research: 242 papers and 675 citations. The claim gained unfounded authority through citation bias against the papers that refuted or weakened it, through papers that repeated it without presenting new data, and through hypotheses turned into apparent facts by citation alone.
• Duyx and colleagues (2017) pooled 52 studies of citation bias. Articles with statistically significant results were cited about 1.6 times as often as articles with nonsignificant results, and articles whose authors concluded that their hypothesis was supported about 2.7 times as often. The bias was most evident in the biomedical sciences and least in the natural sciences.