Recall bias

The infection during pregnancy

Researchers studying the causes of a birth defect interview mothers after delivery. Mothers of affected babies report having had a urinary tract infection during pregnancy noticeably more often than mothers of healthy babies. The researchers conclude that such infections may raise the risk.

The clear-cut case. A mother whose baby has a birth defect has spent months asking herself what might have caused it and is likely to remember a minor infection; a mother of a healthy baby may simply have forgotten one. When Martha Werler and colleagues checked mothers’ interview answers against records made during their pregnancies, mothers of malformed babies were considerably more accurate reporters of urinary tract and yeast infections than mothers of other babies, but equally accurate about using over-the-counter drugs (see Evidence). The same study can be biased for one exposure and not another.

After the newspaper story

A local paper runs a story suggesting that a new rubber surface at a town’s playgrounds is causing children’s skin rashes. A health department survey then asks parents how often their children played there. Parents of children who had rashes report many more visits than parents of children who didn’t.

The difference here may come less from the rash than from the story. Parents who had read that the surface was a suspected cause, and whose child had a rash, had a reason to think hard about playground visits and to count generously. In a Canadian study of childhood leukemia, Infante-Rivard and Jacques found that parents of cases from one area, where an excess of the illness had been publicly attributed to power lines, were far more likely to report correctly that they lived close to power lines, while elsewhere cases and controls recalled about equally, and poorly (see Evidence). Public concern can create recall bias where the condition alone did not.

The snacks nobody remembers

A dental study asks teenagers with several cavities and teenagers with none how often they ate sugary snacks over the past year. The two groups report similar amounts, and the study finds no link.

Recall bias doesn’t only create associations; it can hide them. If teenagers with cavities, who have been told off about sugar at the dentist, tend to underreport snacks more than those without, their reported intake is pushed down toward the other group’s. The Catalogue of Bias notes that undesirable habits tend to be underreported and that recall bias can increase or decrease the strength of an association.

• Cases recall more: the condition prompts a search for causes, as with the birth defect study.
• Recall shaped by publicity or beliefs: what people have heard about a suspected cause changes how they remember their exposure, as with the playground.
• Underreporting by one group: embarrassment or guilt about a habit makes one group report less of it.
• Recall shaped by the outcome’s later effects: McIntosh and colleagues found that mothers of people with schizophrenia, and of people at high risk of it, recalled more obstetric complications (problems in pregnancy and delivery) than mothers of healthy volunteers, although hospital records showed no difference. In the high-risk group, the number recalled was related to the child’s abnormal behavior in childhood as rated by the mother.

Not the same: inaccurate recall. Most memories of the past are imperfect. Recall bias is the case where accuracy differs between the groups being compared. Steven Coughlin stresses that “a distinction must be drawn between recall which is biased and that which is simply inaccurate.”

• The exposure was recorded before the outcome was known, as in a prospective study that collects information when people join, or a study that uses records made at the time. The Catalogue of Bias calls a prospective design the most appropriate way to avoid recall bias.
• The groups recall equally well, which can be checked by comparing what people report with records for a subset of them.
• The controls have the same reason to search their memories. Werler and colleagues advised using mothers of babies with other malformations as controls: those mothers have a similar reason to look back.
• The exposure doesn’t depend on memory at all, such as one measured from records, samples or documents.
• Participants don’t know the hypothesis, and the questions are asked the same way of everyone, so there’s less to steer recall toward one exposure.

The test: could the outcome, or what people believe about it, have changed how well one group remembers the exposure?

Equally hazy memories

A study of people with and without a back injury asks each how many hours a week they spent lifting heavy objects ten years earlier. A check against old workplace records shows both groups got it wrong about as often, and in both directions.

Everyone’s memory was poor, so someone might dismiss the study as “recall bias”. But the errors were similar in both groups, so they don’t create a difference between them. Inaccurate recall still makes the comparison less precise, but it’s the simply inaccurate recall Coughlin distinguishes from the biased kind. That’s the groups recall equally well condition, checked rather than assumed.

Records from before the outcome

A study of a childhood condition compares children who developed it with children who didn’t, using the health visits, infections and medications recorded in their files during their first two years, long before anyone knew which children would develop it.

It’s a comparison of cases and controls about the past, the classic setting for recall bias, and a reader might raise the objection by reflex. But nobody was asked to remember anything: the information was written down before the outcome existed, so it couldn’t have been shaped by it. That’s the recorded before the outcome condition. The records can have their own gaps, but not this one.

Memory isn’t a recording. The Catalogue of Bias notes that “the accuracy and volume of memories may be influenced by subsequent events and experiences”, and that recall depends on how long ago the events were, on people’s age and education, on how important the condition is to them, and on what they already believe. An illness or a child’s birth defect is exactly the kind of later event that sends people back through the past, looking for causes. Coughlin’s review adds that recall bias may be more likely when recall of an exposure is poor in general, which leaves more room for motivation to fill the gaps.

Knowing how things turned out also colors how the past is remembered, a tendency that resembles Hindsight bias. The researcher’s side of the interview can add to it: an interviewer who knows who is a case and expects a certain answer may probe harder, a form of the Observer-expectancy effect. Case-control studies are also vulnerable to choosing controls who differ from cases in other ways, a separate problem described under Selection bias.

• Ask when the information was collected. Was the exposure recorded before the outcome was known, or reported from memory afterward?
• Ask whether recall was checked. Studies sometimes compare interview answers with records for some participants, which shows whether the groups remembered differently.
• Look at the controls. Did they have a comparable reason to think back, such as a different condition?
• Check whether the hypothesis was hidden and the questions standardized. The Catalogue of Bias stresses asking every participant in the same way so as not to influence answers.
• Ask whether publicity preceded the study. A widely reported suspected cause can change what people remember.
• Don’t assume a case-control finding is wrong. Recall bias is a reason to look for confirmation from prospective data, not proof that the association isn’t real; some studies that checked for it found little or none.

Recall bias is a flaw in method rather than an effect with a replication record, but validation studies that compare what people remember with records made at the time have measured it directly. Their results are mixed, which is itself the main finding: recall bias is real in some circumstances and for some exposures, not a fixed property of case-control studies.

• Werler, Pober, Nelson and Holmes (1989) compared postpartum interviews with obstetric records for mothers who delivered at a Boston hospital in 1984. Their measure, relative sensitivity, is how accurately mothers of malformed infants reported an exposure compared with mothers of nonmalformed infants (above 1.0 means the mothers of malformed infants were more accurate). It was 2.7 for urinary tract or yeast infection, 1.2 for antibiotic or antifungal drug use and 1.0 for any over-the-counter drug use, among the eight exposures studied. They concluded that recall bias was present for some exposures and advised using mothers of babies with other malformations as controls.
• Mackenzie and Lippman (1989) questioned women at a Montreal hospital twice, early in pregnancy and after delivery, about 39 exposures. Changes between the two reports were similar among mothers with good, intermediate and poor pregnancy outcomes, were unrelated to maternal concern about the baby, and didn’t importantly change the estimated odds ratios. They found no evidence of biased reporting.
• Infante-Rivard and Jacques (2000) checked parents’ reports within a case-control study of childhood leukemia in Quebec. For distance from home to power lines, compared with measured distance, the sensitivity of parents’ reports (roughly, the share of truly close homes reported as close) was 62% for cases from an area where an excess of the illness had been perceived and attributed to power lines, against 35% for other cases, 36% for hospital controls and 22% for population controls. For prenatal X-rays, recall was similar for cases and population controls. They concluded that recall can be differential under special circumstances, such as heightened public concern, but is otherwise most often nondifferential, with low sensitivity.
• McIntosh and colleagues (2002) found that mothers’ recall suggested more birth complications among people with, or at high risk of, schizophrenia, while records made at the time showed no difference between the groups.
• Coughlin (1990), reviewing the literature, found that the extent of inaccurate recall depends on the exposure and on the respondents, and that interviewing technique, questionnaire design and the study protocol, which investigators control, play a central role.

What remains uncertain is how often recall bias meaningfully changes the conclusions of case-control studies, since that depends on the exposure, the population and the study, and relatively few studies have been able to check recall against records.