Category
Research biases
A research bias is a systematic error in the process of finding things out: in how a study chooses who or what to look at, how it measures, how it analyzes the data, or which results get reported. The epidemiologist David Sackett, who compiled one of the first catalogs of them in 1979, described biases that “may distort the design, execution, analysis, and interpretation of research.”
The key word is systematic. Random error makes results noisy, and more data averages it away. Bias pushes results in a consistent direction, so more data of the same kind just makes the wrong answer more precise.
Entries (0)
Nothing here yet.
Two meanings of “bias”
This site uses “bias” in two senses:
- A Cognitive bias is a tendency in a person’s judgment.
- A research bias is a flaw in a method. It can distort results even when every researcher involved is careful, honest and unbiased in the first sense.
The two often meet. Confirmation bias in a researcher can lead to a biased analysis, which is why good methods are designed to work even when the people using them aren’t neutral.
The main families
Epidemiology textbooks group systematic error into three broad kinds, and evidence-based medicine adds a fourth about what gets published:
- Selection bias: the people or things studied differ systematically from the ones the conclusion is about. Surveying only customers who stayed tells you little about why others left.
- Information (measurement) bias: the way something is measured or recorded differs systematically between groups, such as people who know they’re in the treatment group reporting their symptoms differently.
- Confounding: a third factor drives both the supposed cause and the effect, so a real association gets mistaken for causation.
- Reporting and publication bias: results that are positive, striking or statistically significant are more likely to be written up and published, so the published record overstates how often things work.
Researchers’ own choices can create bias too. Questionable research practices, such as trying many analyses and reporting the one that “worked” (p-hacking) or presenting a hypothesis formed after seeing the results as if it had been predicted (HARKing), are a recognized source of distortion, and are often not deliberate.
Bias isn’t the same as fraud
Most research bias involves no dishonesty at all. Fraud is fabricating or falsifying results. Bias is a structural tilt that honest people fall into unless the study is designed to prevent it, through randomization, blinding, preregistration or publishing null results.
It’s not only a scientist’s problem
The same flaws show up whenever anyone learns from examples: judging a career path by the people who succeeded at it, rating a remedy by the testimonials of people it helped, or deciding a neighborhood is dangerous from the incidents that made the news. Knowing these patterns helps you read a headline, a review or a success story, not just a journal article.
A related idea about claims rather than methods is Unfalsifiability: a hypothesis that no possible result could count against isn’t made more reliable by any amount of research.
Other categories: Formal fallacies · Informal fallacies · Cognitive biases