Category
Research biases
A research bias is a systematic error in the process of finding things out: in how a study chooses who or what to look at, how it measures, how it analyzes the data, or which results get reported. The epidemiologist David Sackett, who compiled one of the first catalogs of them in 1979, described biases that “may distort the design, execution, analysis, and interpretation of research.”
The key word is systematic. Random error makes results noisy, and more data averages it away. Bias pushes results in a consistent direction, so more data of the same kind just makes the wrong answer more precise.
Entries (14)
- Collider biasA spurious link between two things that appears when data are restricted to, or adjusted for, something both of them affect.
- ConfoundingA third factor that affects both an exposure and an outcome makes the association between them a misleading guide to the exposure’s effect.
- Ecological fallacyAssuming that a relationship seen across groups, such as states or schools, also holds for the individuals within those groups.
- Funding biasResearch paid for by an organization with a stake in the answer tends to reach answers that favor it, through choices in design, analysis and reporting.
- Observer-expectancy effectA researcher’s expectations shape what they record or how subjects behave, so the study partly produces the result it was meant to test.
- P-hackingTrying different ways of collecting or analyzing data until a result comes out statistically significant, then reporting that version as the finding.
- Publication biasStudies with positive or statistically significant results are more likely to be published, so the published record makes effects look stronger than they are.
- Recall biasWhen groups in a study remember their past with different accuracy, so a difference in what they report reflects memory, not what actually happened.
- Regression fallacyContestedGiving the credit or blame for a change to something that happened in between, when an unusual result was simply returning toward average.
- Selection biasA distortion that arises when the cases studied differ systematically from the ones a conclusion is about, because of how they got into the data.
- Simpson's paradoxA trend that holds in every subgroup reverses or vanishes when the groups are combined, because the groups differ in size and makeup.
- Texas sharpshooter fallacyTreating a pattern in data as meaningful when the boundaries that make it stand out were drawn after seeing where the data fell.
Two meanings of “bias”
This site uses “bias” in two senses:
- A Cognitive bias is a tendency in a person’s judgment.
- A research bias is a flaw in a method. It can distort results even when every researcher involved is careful, honest and unbiased in the first sense.
The two often meet. Confirmation bias in a researcher can lead to a biased analysis, which is why good methods are designed to work even when the people using them aren’t neutral.
The main families
Epidemiology textbooks group systematic error into three broad kinds, and evidence-based medicine adds a fourth about what gets published:
- Selection bias: the people or things studied differ systematically from the ones the conclusion is about. Surveying only customers who stayed tells you little about why others left.
- Information (measurement) bias: the way something is measured or recorded differs systematically between groups, such as people who know they’re in the treatment group reporting their symptoms differently. Recall bias and the observer-expectancy effect are examples.
- Confounding: a third factor drives both the supposed cause and the effect, so a real association gets mistaken for causation.
- Reporting and publication bias: results that are positive, striking or statistically significant are more likely to be written up and published, so the published record overstates how often things work.
Researchers’ own choices can create bias too. Questionable research practices, such as trying many analyses and reporting the one that “worked” (P-hacking) or presenting a hypothesis formed after seeing the results as if it had been predicted (HARKing), are a recognized source of distortion, and are often not deliberate.
Bias isn’t the same as fraud
Most research bias involves no dishonesty at all. Fraud is fabricating or falsifying results. Bias is a structural tilt that honest people fall into unless the study is designed to prevent it, through randomization, blinding, preregistration or publishing null results.
It’s not only a scientist’s problem
The same flaws show up whenever anyone learns from examples: judging a career path by the people who succeeded at it, rating a remedy by the testimonials of people it helped, or deciding a neighborhood is dangerous from the incidents that made the news. Knowing these patterns helps you read a headline, a review or a success story, not just a journal article.
A related idea about claims rather than methods is Unfalsifiability: a hypothesis that no possible result could count against isn’t made more reliable by any amount of research.
Other categories: Formal fallacies · Informal fallacies · Cognitive biases