Overconfidence

The quiz night ranges

At a trivia night, each team gives a range for ten questions (“the length of the Danube”, “the year the first ballpoint pen was patented”), wide enough to be 90% sure of containing the answer. One team’s ranges catch the right answer four times out of ten.

If the team were calibrated, about nine of ten ranges would include the answer. Four out of ten means the ranges were drawn around what the team felt sure of, not around how much they actually knew. This is the laboratory task behind most of the research, and results like this are typical: Moore and Healy’s review notes that 90% ranges usually contain the answer less than half the time.

The sales forecast

Each month, a bakery manager predicts next month’s bread sales and adds, “almost certainly between 1,900 and 2,100 loaves.” Over the year, actual sales fall inside that range in five months out of twelve. The best guess in the middle is usually close.

This shows that overprecision is a separate problem from a bad estimate. The manager’s central forecasts are reasonable; it’s the stated certainty around them that is wrong. Anyone who orders flour or schedules staff on the promise of “almost certainly” will be caught short in the months that fall outside the range.

The experienced eye

A carpenter with twenty years’ experience cuts boards by eye and says, “I’m always within a quarter inch.” When an apprentice measures a week’s worth of cuts, most are that close, but about a third are off by half an inch or more.

Experience made the carpenter more accurate, but not accurately confident. Research finds overprecision in experts as well as novices. Skill narrows your errors; it doesn’t automatically teach you how wide they still are.

Don Moore and Paul Healy distinguish three things studied under the same name. They behave differently, so this entry’s status applies to the first only.

• Overprecision: too much certainty that you know the truth. Robust (see Evidence).
• Overestimation: thinking your performance, ability or chances are better than they are (“I got at least eight of these right”). Common on hard tasks but reversed on easy ones, where people underestimate themselves. Moore and Schatz call the evidence for general overestimation “thin and inconsistent”. They name the Planning fallacy and the Illusion of control as the two literatures where overestimation is reported consistently, and note reversals in both.
• Overplacement: thinking you’re better than others. The better-than-average effect is very reliable as a pattern of self-ratings, but whether it is an error is disputed, and on hard tasks people often underplace themselves. The Dunning–Kruger effect concerns this kind of comparison.

• When your confidence matches your hit rate. A narrow range is fine if ranges like it have actually contained the answer about as often as you claim.
• When one confident judgment turns out wrong. Being 90% sure means being wrong about one time in ten. A single miss says nothing; overconfidence is a pattern across many judgments.
• When you gave a best guess, not a certainty. A point estimate doesn’t claim any precision; the error is in the confidence attached to it.
• When the question really is easy for you. High confidence about things you reliably know is just accurate.

The test: across many judgments like this one, how often have I been right when I felt this sure?

A tight range with a log behind it

A commuter says, “My drive to work takes 25 to 30 minutes, nine days out of ten.” They have tracked it with an app for six months.

The range is narrow and stated with confidence, but it comes from a record of outcomes that shows it captures about nine days in ten. That’s calibration, which is exactly what overconfidence lacks.

The 90% forecast that missed

A weather service said there was a 90% chance of rain for a town’s summer fair. It stayed dry, and a councilor complains that the forecasters are overconfident.

One miss can’t show this. If, across all the days the service has forecast a 90% chance of rain, it has in fact rained on about nine in ten, the forecasts are well calibrated and this was one of the expected dry days. Judging the forecast by a single result is closer to Hindsight bias than to detecting overconfidence.

• Ranges built from too little of what you know. Jack Soll and Joshua Klayman found that subjective intervals are systematically too narrow for the accuracy of people’s knowledge, sometimes only about 40% as wide as calibration would need. Variability in how people set their widths adds to the problem.
• Treating a small sample as the whole picture. Peter Juslin and colleagues’ “naïve intuitive statistician” model proposes that people accurately describe the few relevant cases they can call to mind but treat that small sample’s spread as if it were the spread in the world. Small samples understate variability, so the ranges come out too narrow. This also explains why producing a range leads to more overconfidence than judging the probability of a range someone else gives you.
• Not imagining the alternatives. When people are made to consider the whole range of possible answers, overprecision shrinks (see How to respond), which suggests ordinary judgments focus too narrowly on the answers that first come to mind. This relates to “what you see is all there is”.

Overestimation and overplacement have a different explanation in Moore and Healy’s account: people are uncertain about their own performance and even more uncertain about others’, so their estimates are pulled toward the middle. That produces overestimation on hard tasks, underestimation on easy ones, and the opposite pattern for placement.

• Keep score. Write down confident predictions with a stated probability and check them later. Calibration is a record, not a feeling. (This follows from how overconfidence is defined; it hasn’t been tested here as a standalone remedy.)
• Spread probability over the whole range. Uriel Haran, Don Moore and Carey Morewedge divided the full range of possible answers into bins and asked people how likely each bin was. Across three experiments, this reduced overprecision, and it carried over: people later gave wider conventional ranges too.
• Ask for the ends separately. Asking separately for a value you’re 90% sure the answer is above, and one you’re 90% sure it’s below, rather than for a single range, modestly reduced overprecision in Soll and Klayman’s studies, as Haran and colleagues summarize them.
• Judge a range instead of producing one. Research reviewed by Haran and colleagues found that rating how likely a given range is to contain the answer produces less overconfidence than making up a range yourself, although in one study the improvement didn’t carry over to later judgments in the usual format.

Status: replicates robustly, for overprecision, which is the core this entry defines. Overprecision has been found by many independent groups for decades, including researchers who dispute other overconfidence findings, and in a preregistered study of about 1,700 participants in four countries. The status does not cover overestimation or overplacement, whose status as errors is disputed (see below); an entry defining overconfidence as “thinking you’re better than you are” would be contested.

Overprecision.

• Soll and Klayman (2004) asked people for high and low estimates they were a given percent sure contained the answer to general-knowledge questions. They showed “substantial overconfidence”: the answer fell inside the intervals much less often than the stated confidence. Choosing between two possible answers showed much less overconfidence. How much overconfidence appeared depended strongly on how the intervals were asked for, and on the domain.
• Moore and Healy (2008) reviewed the literature, noted that 90% confidence intervals typically contain the right answer less than half the time, and concluded that overprecision “appears to be more persistent” than the other two kinds. Different ways of asking change how much overprecision appears, but do not produce underprecision.
• Juslin, Winman and Hansson (2007), whose group has argued that much two-choice overconfidence is an artifact (below), treat overconfidence in producing intervals as a robust finding and offer a model to explain it.
• Haran, Moore and Morewedge (2010) describe overprecision as “the most robust type of overconfidence”.
• Moore, Dev and Goncharova (2018) ran a preregistered study across the US, UK, India and Hong Kong (about 1,700 participants after exclusions; data from two countries had been collected before preregistration). Participants’ 90% intervals for the area of a shape contained the right answer only 31% of the time. Item-by-item confidence ratings also exceeded accuracy.

The main caveats are about size and measurement, not existence. Moore and Schatz (2017) point out that most evidence comes from a few paradigms using questions people rarely face in daily life, that ordinary people are unfamiliar with confidence intervals, and that item-confidence measures can exaggerate the effect. They call overprecision “the most pervasive but least understood form of overconfidence”. Much of the recent preregistered work comes from Moore’s lab.

Overestimation: disputed. The hard–easy effect (overestimation on hard tasks, underestimation on easy ones) complicates any general claim. Juslin, Winman and Olsson (2000) reviewed two-choice general-knowledge studies and found “very little support for a cognitive-processing bias”. Results differed between questions hand-picked by researchers and questions sampled representatively, in a way difficulty didn’t explain, and the hard–easy effect nearly vanished once statistical artifacts were controlled. Moore and Schatz conclude that “the evidence for overestimation is actually quite weak”.

Overplacement: the pattern is robust, the error is disputed.

• Zell and colleagues (2020) meta-analyzed 291 samples with more than 950,000 participants and found a robust better-than-average effect (d = 0.78) with little evidence of publication bias, larger for personality traits than for abilities.
• Koppel and colleagues (2023) ran a preregistered replication (1,203 participants) of Ola Svenson’s 1981 driving study, in which most people rate their driving skill and safety above that of the other participants, and found the same result on three different response scales.
• But a majority rating themselves “above average” isn’t necessarily an error when abilities are skewed or people define “good” differently. Moore and Schatz argue that among studies avoiding these problems overplacement shrinks, and underplacement is common on difficult tasks: people expect to be worse than others at juggling or at winning hard competitions. Moore and Healy’s experiment showed overplacement on easy quizzes and underplacement on hard ones.

What remains uncertain. Why overprecision happens, how large it is outside the laboratory tasks used to measure it, and whether it’s a stable trait of individuals. Structured settings can shrink it a lot: in a three-year geopolitical forecasting tournament designed with this research in mind, Moore and colleagues (2017) found forecasters’ confidence roughly matched their accuracy, with about 3% overconfidence, reduced to 1% by training and teamwork. For overestimation and overplacement, the open question is how much is a genuine bias rather than noisy judgment pulled toward the middle.