Unfalsifiability

The same kind of claim, from testable to untestable.

A bold claim

“Drink this cleanse every morning for two weeks and your afternoon headaches will stop.”

This claim forbids something: afternoon headaches in week three. If they keep coming, the claim is in trouble. That risk is what makes it worth listening to, and what would make it impressive if it held up.

The same claim, rescued

The headaches continue. “You must have had some caffeine. The cleanse only works if you follow the plan exactly.” After a caffeine-free month with no change: “Headaches that continue are a sign the toxins are leaving. It means it’s working.”

Each excuse arrived only after a failure, and none can be checked on its own: nobody says what “exactly” means, which toxins, or how you’d tell toxins leaving apart from a cleanse that does nothing. Now every result, including the one the original claim forbade, counts as support. The claim started out falsifiable and became unfalsifiable in practice.

Unfalsifiable from the start

A weekly horoscope: “Someone close to you may test your patience, but an unexpected opportunity is within reach.”

There’s no week in which this could turn out wrong. Almost everyone has a frustrating moment and something that could be called an opportunity. Popper made this point about astrology: by keeping its predictions vague enough, it could explain away anything that would otherwise have counted against it.

A useful claim about the world rules something out. “The library opens at 9” forbids finding it locked at 9:30, and that’s what makes it worth knowing. Popper put it this way: every good scientific theory “forbids certain things to happen”, and the more it forbids, the better it is.

A claim compatible with every outcome forbids nothing, so it can’t help you plan, choose or predict. It can still feel constantly confirmed:

“When things go wrong, the universe is testing me. When they go right, it’s rewarding me for passing.”

Every day brings more “confirmation”, but a bad day and a good day both fit, so neither tells you anything. Popper noticed the same pattern in the psychological theories of Freud and Adler, which he argued could interpret any human behavior at all: their admirers saw confirming cases everywhere, and he came to think this apparent strength was really a weakness. Confirmations should count, he argued, only when they come from a risky prediction, one that could have failed.

A claim doesn’t have to start out unfalsifiable. Often a testable claim fails a test and is saved by an ad hoc addition: an extra assumption brought in only to explain away that failure, or a reinterpretation of the claim so that it no longer conflicts with the result. Popper called this a “conventionalist stratagem”. It’s always possible, he wrote, but it rescues the theory only by destroying, or at least lowering, its scientific status.

Not every adjustment after a failure is a rescue, though. A test never checks a claim entirely on its own: it also relies on the instruments, the conditions and whether the procedure was followed, so “maybe something else went wrong” is often a reasonable thought. A practical way to tell the two apart:

The philosopher Imre Lakatos drew a similar line for whole research programs: progressive ones, whose revisions make surprising predictions that are then confirmed, and degenerating ones, whose revisions are made only to accommodate facts already known.

In everyday argument, rescue moves have names of their own. “No true Scotsman” protects a generalization from a counterexample by redefining the group so the counterexample doesn’t count. “Moving the goalposts” changes what would count as success or failure after the original standard has been met.

Popper proposed falsifiability to separate empirical science from everything else, not as a test of whether a statement means anything. In his words, it “draws a line inside meaningful language, not around it.” Plenty of claims are unfalsifiable without being defective:

• Mathematics and logic. No observation could refute “7 + 5 = 12”. If you count seven apples and five more and get eleven, you miscounted. That isn’t a failed empirical claim; it isn’t an empirical claim at all.
• Definitions. “A bachelor is an unmarried man” is true because of what the words mean, not because of a survey.
• Many philosophical and ethical claims. “Cruelty for its own sake is wrong” isn’t settled by an experiment. Claims like this are argued for with reasons, not tested by observation.
• Tautologies. Popper’s own example: “It will rain or not rain here tomorrow” can’t be refuted, so it isn’t empirical, but it isn’t nonsense either.

Popper didn’t treat the theories he called untestable as worthless, either. He wrote that calling them untestable didn’t mean Freud and Adler had seen nothing correctly, and that ideas which begin as untestable can later be developed into testable ones.

So “that’s unfalsifiable” doesn’t show a claim is false (see Valid vs. true). The real question is whether the claim is being offered as a factual claim about the world, backed by evidence. The cleanse seller claims results in your body, and the horoscope claims to describe your week. A claim that can’t lose to any evidence can’t be supported by the evidence it cites either.

Falsification has the shape of modus tollens:

If the theory is true, we’ll observe X. We don’t observe X. So the theory is false.

Popper built his proposal on an asymmetry. No number of passing tests can prove a universal claim such as “all copper conducts electricity”, but a genuine counterexample can refute it by purely deductive reasoning, “with the help of the modus tollens of classical logic”, as he put it. The reverse inference, “we observed X, so the theory is true”, has the form of Affirming the consequent, which is why passing a test never proves a theory.

In practice, though, the “if” part is never the theory alone. That’s where the criterion gets harder to apply (see Limits).

• Argument from ignorance: “You can’t disprove it, so it’s true” treats immunity to disproof as a point in a claim’s favor. For an unfalsifiable claim, failing to disprove it is guaranteed, so it’s no evidence at all.
• Confirmation bias: favoring the cases that fit what you already believe. An unfalsifiable claim makes every case fit.
• Hindsight bias: once you know what happened, a vague claim is easy to read as having predicted it.
• In research (Research bias): Nosek and colleagues describe a common case where “the researcher’s prediction is vague so that many possible outcomes can be rationalized after the fact as supporting the prediction”. Forming a hypothesis after seeing the results and then evaluating it on the same data (hypothesizing after the results are known, or HARKing), which they call circular reasoning, has a similar effect: those data could never have counted against it. Preregistration, which fixes the prediction and the analysis before the data come in, is one response.

The practical question is “What would count against this?”, asked before the result is in. If the honest answer is “nothing”, the claim can’t be supported by evidence either, whatever else it may be. If a result that should have counted against a claim arrives and the claim gains a new excuse, ask whether the excuse can be checked on its own, or exists only to absorb that result.

Falsifiability is the best-known criterion for telling science from pseudoscience, but philosophers of science don’t treat it as settled:

• Most pseudosciences have been tested and failed. Hansson points out that astrology, which Popper took as a clear example of pseudoscience, “has in fact been tested and thoroughly refuted”, and that the main threats to psychoanalysis’s scientific status come from claims that it failed its tests, not that it can’t be tested. Popper’s reply was that a doctrine loses scientific status when its promoters break “the methodological rule that we must accept falsification”.
• No test checks a claim alone. A prediction follows only from a theory together with auxiliary assumptions about instruments and conditions. When it fails, logic says something in the bundle is wrong, but as the physicist Pierre Duhem put it, the experiment “does not designate which one should be changed”. The philosopher W. V. O. Quine extended the point to all our beliefs; together this is often called the Duhem–Quine thesis. So a failed test can always, logically, be blamed on something other than the theory. Popper conceded that falsification can always be evaded this way, and responded by making the refusal to evade it part of what defines empirical method, which means deciding when a rescue is ad hoc takes judgment, not logic alone.
• Other accounts. Thomas Kuhn argued that most science is “normal science”, solving puzzles within an accepted theory rather than trying to refute it, and proposed puzzle-solving as the mark of science: on his view astrology was never a science because its failures didn’t produce research puzzles anyone could use to improve it. Lakatos proposed judging research programs over time rather than single theories at a single moment. Many later authors use a list of several criteria instead of one, on which testing “so arranged that the theory can only be confirmed” appears as one warning sign among several.

So “that’s unfalsifiable” is a good question to put to a claim that presents itself as empirical, not a finished verdict that the claim is pseudoscience, or that it’s false.