Hasty generalization

Two drills

“The last two cordless drills I bought from that brand both died within a year. Their tools are junk.”

The clear-cut case. A brand makes dozens of tools, built on different lines and bought for different jobs, and failure rates vary. Two failures, possibly both used the same hard way, can’t tell you how that brand’s tools do in general. The speaker has a reason for caution, and perhaps for a closer look at reliability data; they don’t have grounds for “junk”.

A big poll that answered itself

A hiking forum runs a poll: “Would you pay a $10 fee to keep state trails maintained?” 4,000 members respond, and 85% say yes. A post announces: “Most people in the state would pay for trail access.”

This isn’t a small sample, and that’s the point. The respondents are members of a hiking forum who chose to answer a poll about trails, so they almost certainly care more about trails than the average resident. A sample picked like this is self-selected, a form of selection bias (see Research bias): adding more responses of the same kind makes the wrong answer more precise, not less wrong. The poll is good evidence about keen hikers, not about the state.

The right sample, too big a claim

A school randomly selects 40 of its 900 students for a short survey. 30 of them say the lunch lines are too long. The principal’s newsletter reports: “Students agree: the lunch lines are too long.”

The sampling is sound, and “most students think the lines are too long” is a reasonable conclusion from 30 out of 40 chosen at random. But the newsletter’s wording, a statement with no “most” or “some”, reads as if it applies to students in general, and 10 of the 40 didn’t agree. The error is in the size of the claim, not the sample. Researchers call unquantified statements like “students agree” or “dogs are loyal” generics; they are easy to read as “all”, and can hide real disagreement.

The missing premise is where the argument is won or lost, and it’s a factual question about the group and how the sample was drawn, not a matter of logic. Uwe Peters and colleagues, reviewing hasty generalizations in medical research, describe them in these terms: a conclusion is extended from a sample to a broader population when the sample isn’t large or diverse enough, or when there is no support for thinking the sample and the population are similar enough to justify the step.

• Too few cases for a varied group: a couple of encounters, reviews or trials taken to settle a question about something that varies a lot.
• An unrepresentative sample: the cases are numerous but chosen in a way that tilts them, such as volunteers, survivors, people who complained, or whoever was easiest to reach.
• Anecdotal evidence: one story, often vivid or personal, standing in for the whole group. “My grandfather smoked and lived to 95” is a generalization from a sample of one, usually offered against much larger evidence.
• Overreaching claims: a sample that supports “some” or “many” is reported as “all”, “always” or an unquantified generic.
• Beyond the population sampled: findings from one population (one country, one age group, one season) stated as if they held everywhere. Peters and colleagues found this pattern common in published research.

The error is old and has had other names. Isaac Watts’s Logick (1724) listed false induction, the mistake of generalizing on insufficient evidence, and John Stuart Mill’s System of Logic (1843) grouped errors of this kind among its fallacies of generalization, one of two families of errors in inductive reasoning.

Not the same error: the fallacy of composition. Composition reasons from the parts of one thing to the whole thing (“every part of this machine is light, so the machine is light”). A hasty generalization reasons from some members of a group to the other members. Ignoring qualifications (secundum quid), applying a rule that holds in general as if it had no exceptions, is also closely related; Douglas Walton treats it as a subtype of hasty generalization.

Generalizing from a sample is how most knowledge about the world is gained, and a small sample is often plenty. It holds up when:

• The group is uniform in the way that matters. One spoonful of a well-stirred soup, one sample from a thoroughly mixed batch of paint, one measurement of the boiling point of pure water at sea level: when there is good reason to expect little variation, very few cases are needed. The reason has to come from background knowledge, though. Assuming a group is uniform because you’ve only seen a few of its members is the error itself.
• The sample was chosen so it’s likely to resemble the group. Random selection is the standard way; so is deliberately covering the ways the group varies. Size helps only once the choosing is fair.
• The conclusion claims no more than the sample can show. A single case is enough to show that something can happen (“at least one”). A few cases support “some”. “Most” needs a representative sample. “All” or “never” needs strong reason to expect uniformity, and one counterexample defeats it.
• The conclusion is held as provisional. A tentative working assumption from limited experience (“this café is usually quiet in the mornings”), revised as more cases come in, is sensible.

The test, in three questions: how much does this vary across the group, how were these cases picked, and how much am I claiming?

One taste

A cook stirs a large pot of soup, tastes one spoonful and says, “It needs more salt.”

One spoonful out of perhaps two hundred in the pot, and the conclusion is about the whole pot. It’s sound because stirring makes the pot uniform: salt dissolved in liquid spreads evenly, so any spoonful is very likely to taste like every other. That’s the uniform in the way that matters condition. The same single taste would say much less about a pot of chunky stew, where one spoonful might be all potato.

“It can be done”

The instructions for a garage-door opener say installation requires a professional. A homeowner says, “My neighbor installed the same model himself last weekend, following the manual, and it works fine. So it’s possible for someone without training to do it.”

A sample of one, but the conclusion is only that it’s possible. A single real case proves that something can happen. That’s the condition that the conclusion claims no more than the sample can show. It would become a hasty generalization if the homeowner concluded “anyone can install these easily”.

People judge from what’s in front of them and rarely ask what’s missing, Daniel Kahneman’s “what you see is all there is”. A few cases, especially vivid or personal ones, form a coherent story, and a coherent story feels like enough. Our own experiences also feel far more informative than a statistic about strangers. When they conflict, the same pull leads people to under-weight information about how common something is in the population, as in Base rate neglect.

Amos Tversky and Daniel Kahneman described a related intuition as a belief in the law of small numbers: people expect even a small random sample to be highly representative of the population it came from, so they trust small samples more than they should. Their examples came from professional psychologists making decisions about research.

People aren’t blind to sampling, though. Richard Nisbett and colleagues found that everyday reasoners use rough statistical principles more often when it’s clear what is being sampled and how, and when the role of chance is obvious. Later work using their method finds that people generalize more readily from a few cases for kinds of things they take to be uniform. That’s the right instinct when the belief is correct, and the source of the error when it isn’t.

• Ask how the cases were found. Were they chosen at random, or are they the ones that volunteered, complained, survived or came to mind? A biased route to the sample isn’t fixed by adding more cases.
• Ask how much the thing varies. If people, products or places differ a lot in this respect, a few cases can’t speak for the rest.
• Match the claim to the sample. Often the fix isn’t to drop the conclusion but to shrink it: “all” to “some”, “people” to “the people we asked”.
• Look for counterexamples before accepting “all” or “never”. One is enough to defeat a universal claim, though not a claim about “most”.

Hasty generalization is a matter of reasoning rather than an empirical effect, but researchers have measured how often it appears in published science.

• Peters and Lemeire (2024) analyzed 171 experimental philosophy studies published between 2017 and 2023. Most tested only Western populations but generalized beyond them without justification. There was no evidence that studies with broader conclusions had larger or more diverse samples, but those studies had higher citation impact.
• Peters, Sherling and Chin-Yee (2024) reviewed 533 prospective studies in four leading medical journals. 52.5% generalized beyond the national populations they had studied, and only 10.2% of those reported factors that could support extending the findings. There was no evidence that articles with broader conclusions had larger or more nationally diverse samples. The authors conclude that many of these generalizations were insufficiently supported.

Both reviews measure a mismatch between claims and samples, not whether the broad claims turned out to be false. Some may well be true; the point is that the studies didn’t show it.