Funding bias

The rival at half strength

A lawn-care company funds a trial comparing its new fertilizer with a competitor’s. Its own product is applied at the recommended rate; the competitor’s is applied at half its label rate, described in the methods as “a standard maintenance dose”. The company’s lawns come out greener, and the summary says the new fertilizer “outperformed the leading brand.”

The clear-cut case. The measurements may be accurate, but the comparison was set up so the rival product couldn’t do its best. The Catalogue of Bias lists giving a competitor’s drug “at a non-optimal dose” among the ways sponsored comparison trials are tilted, and the same move works for any product. The result answers “is our product better than theirs used wrongly?”, which isn’t the question the summary claims to answer.

The question worth funding

A juice producers’ association funds a university study on whether drinking orange juice raises blood levels of vitamin C and antioxidants. The study is well run, finds that it does, and is published with a clear disclosure of the funding. The association’s website cites it under the heading “The science of a healthy breakfast.”

Nothing in the study is wrong, and the result is probably true. The tilt is in which question got asked: a funder with an interest in juice has good reason to pay for research on its nutrients and little reason to pay for research on its sugar. The Catalogue of Bias lists “posing a research question such that the answer is true but misleading” as one route for the bias. Across many such studies, the published record fills up with accurate answers to the favorable questions, and a reader who counts the evidence gets a lopsided picture without meeting a single flawed paper.

Results versus conclusions

A mattress company sponsors a trial of its new mattress for people with lower back pain. On the planned main outcome, pain scores after eight weeks, there’s no meaningful difference from an ordinary mattress. One of six secondary measures, self-rated sleep comfort, favors the new mattress. The abstract concludes that the mattress “may offer benefits for people with back pain, particularly sleep comfort.”

The data are reported, but the conclusion leans further than the results. The main question came out flat, and the one favorable secondary result out of six is the one the conclusion is built around. This is the reporting stage of the bias: Lundh and colleagues found that in industry-sponsored studies, conclusions agreed with the results less often than in other studies (see Evidence).

The Catalogue of Bias describes several mechanisms, which enter at different stages of a study:

• The question. Studying the outcomes likely to favor the product, as in the juice example.
• The comparison. Testing against a weak comparator, or a competitor at the wrong dose, as in the fertilizer example. Lexchin and colleagues name the choice of an inappropriate comparator as one explanation for the association they found.
• The participants. Enrolling people unrepresentative of those who will actually use the product.
• The analysis. Choices about how to analyze the data that favor the product (see P-hacking).
• What gets published. Not publishing results that don’t favor the sponsor, reporting only favorable outcomes, or publishing favorable findings more than once (see Publication bias). Bekelman and colleagues found industry sponsorship was associated with restrictions on publication and data sharing.
• The conclusion. Wording that goes beyond or around the results, as in the mattress example.

Not the same error: the genetic fallacy. Funding bias is a claim about how studies with a particular kind of funder tend to turn out, measured by comparing many studies. Rejecting one particular result only because of who paid for it, without pointing to any choice that favored the funder, is the genetic fallacy. Funding is a sound reason to examine a study more closely; it doesn’t do the examining.

A study with an interested funder isn’t biased by that fact alone. Its result deserves the same weight as any other when:

• The funder had no control over the study. The Catalogue of Bias recommends that researchers keep control over design, conduct, analysis and reporting, while noting that this may not be enough on its own, since the questions a sponsor chooses to fund can still shape the results. It describes a “firewall”, in which companies contribute to a general research fund but don’t sponsor specific trials, as likely the most effective safeguard.
• The choices where the bias usually enters can be checked, and they hold up. The comparator is the best available alternative at a proper dose, the outcomes were registered in advance and all of them are reported, and the conclusion matches the results.
• Independent studies find the same thing. A result reproduced by researchers without the stake doesn’t depend on the sponsor’s choices.
• The comparison between funders is fair. Sponsored studies can come out more favorable for reasons other than bias, for instance because companies tend to test products that earlier research suggested would work (see the look-alike below).

Industry funding also has real strengths: private companies provide by far the most money for research and development, as Holman and Elliott note, and in Lundh and colleagues’ review sponsored studies scored no worse on most standard checks of trial quality.

The test: at which choices in this study could the funder’s interest have entered, and were those choices made or checked by someone without a stake?

The trial built to be checked

A reader cites a trial of a new hearing aid that found it improved speech understanding in noisy rooms. A colleague dismisses it: “The manufacturer paid for it.” The reader points out that the trial was registered with its outcomes before it started, compared the aid with the best-selling competitor at its recommended settings, reported every registered outcome, and was followed by a university-funded trial with similar results.

The colleague’s concern is a fair starting point, but the reader has done the checking it calls for. The places where a sponsor’s interest usually enters, the comparator, the outcomes and the reporting, were fixed in advance or open to inspection, and an independent study agreed. That’s the choices can be checked and independent studies conditions. Dismissing the result now would rest on the funder alone.

Seeds that were already promising

An agricultural researcher notices that trials of new wheat varieties funded by seed companies report higher yields more often than trials funded by a government program. Before calling it funding bias, she checks how varieties reach each kind of trial: the companies field-test only varieties that did well in years of their own greenhouse screening, while the government program tests a broad sample of experimental lines. The trial methods are otherwise the same.

A difference in results by funder is what funding bias predicts, but it isn’t proof of it. Here there is an explanation that involves no tilt in any individual trial: the companies’ trials start with stronger candidates, so more of them succeed. The researcher’s check is what Krimsky urges: he argues that a “funding effect” is a symptom that several factors could explain, and that bias shouldn’t be the default assumption. That’s the fair comparison condition.

Most of it doesn’t require dishonesty. A funder chooses which projects to pay for. Researchers know that results matter for what gets funded next; the Catalogue of Bias notes that the impact of negative results on future funding opportunities may play a role. And each design or reporting decision usually has a defensible rationale. When several such decisions all lean the same way, the result is tilted even if no one set out to tilt it. People are generally quicker to notice the flaws in evidence that goes against what they want, and slower to notice them in evidence that supports it, a pattern related to confirmation bias.

The bias is also hard to see with the usual tools. Standard checklists of trial quality look for problems such as poor randomization or unblinded assessment. Lundh and colleagues found that sponsored studies did no worse on most of those, and concluded that their findings “suggest the existence of an industry bias that cannot be explained by standard ‘Risk of bias’ assessments.” The tilt tends to live in choices the checklists don’t score: the question, the comparator and the framing of the conclusion.

• Read the funding and conflict-of-interest statements, and notice who had control over the design, data and decision to publish.
• Check the specific choices. Was the comparison with the best alternative, used properly? Were the outcomes registered beforehand, and are all of them reported? Does the conclusion say more than the main result supports?
• Look for independent evidence. Do studies by researchers without the stake point the same way?
• Lower your confidence, don’t reverse it. An undisclosed or tilted study is weaker evidence, not evidence for the opposite. A dismissal that names no questionable choice is the genetic fallacy.

Funding bias is a flaw in method rather than an effect with a replication record, but the association between who pays and what studies find has been measured repeatedly, mostly in medicine.

• Bekelman, Li and Gross (2003) systematically reviewed 37 studies on financial conflicts of interest in biomedical research. Pooling eight of them, which together assessed 1,140 original studies, they found a statistically significant association between industry sponsorship and pro-industry conclusions (pooled odds ratio 3.60, 95% confidence interval 2.63 to 4.91).
• Lexchin and colleagues (2003) reviewed 30 studies. Research sponsored by drug companies was more likely to have outcomes favoring the sponsor (odds ratio 4.05, 95% CI 2.98 to 5.51, 18 comparisons). None of the 13 studies that analyzed methods reported that industry-funded studies were of poorer quality. The authors suggested inappropriate comparators and publication bias as explanations.
• Lundh and colleagues (2017), a Cochrane review of 75 papers, found that industry-sponsored drug and device studies more often had favorable efficacy results (risk ratio 1.27, 95% CI 1.17 to 1.37, 25 papers, moderate-quality evidence) and favorable conclusions (1.34, 1.19 to 1.51, 29 papers, low-quality evidence). There was no difference in risk of bias from randomization, allocation concealment, follow-up or selective outcome reporting, and sponsored studies more often had low risk of bias from blinding. Their results and conclusions agreed less often (risk ratio 0.83, 0.70 to 0.98, six papers).
• Bero and colleagues (2007) examined 192 trials comparing a statin with another drug. Among the industry-funded trials, funding from the maker of the drug being tested was associated with results (odds ratio 20.16) and conclusions (34.55) favoring that drug, with very wide confidence intervals, after controlling for other factors.
• Outside medicine. Lesser and colleagues (2007) classified 206 articles on soft drinks, juice and milk. Among intervention studies, none with all-industry funding had an unfavorable conclusion, against 37% of those with no industry funding. Barnes and Bero (1998) found that of 106 reviews on the health effects of passive smoking, 39 concluded it was not harmful, and 29 of those were by authors with tobacco industry affiliations; controlling for article quality and other factors, affiliation was the only factor associated with that conclusion.

What remains uncertain is how much of the association is bias in the studies themselves. The studies above are observational comparisons of research, and Lundh and colleagues rated much of their evidence moderate or low in quality. Krimsky argues that the “funding effect” is “merely a symptom” of the factors that could produce it, and that social scientists “should not suspend their skepticism and choose as a default hypothesis that bias is always or typically the cause.” Against that, the association has been found consistently, in several independent reviews and in fields from pharmacology to nutrition, and the review with the most papers found that standard quality checks didn’t explain it.