AI Access Collapses 'I Don't Know' Rate From 44% to 3% — Accuracy Drops by Two-Thirds, Confidence Doubles
Three researchers from the University of Milano-Bicocca, École Normale Supérieure, and Sapienza University of Rome ran a controlled experiment to measure what AI access does to the human habit of recognising the limits of one’s own knowledge. The experimental design was adversarial by construction: participants were given access to Step 3.5 Flash, a model specifically chosen because it tends to fail on the questions asked.
The questions were visual film details — things like the colour of a sports team’s uniform in Bend It Like Beckham. On this category, Step 3.5 Flash is typically wrong. That matters for the interpretation: if the model were reliably correct, any accuracy drop could be explained as rational delegation to a better-informed system. In this setup, that explanation is ruled out.
The Numbers
| Condition | IDK rate | Accuracy | Confidence |
|---|---|---|---|
| No AI | 44% | 27% | 30% |
| With AI access | 3% | 9% | 76% |
| With AI + incentives | 8% | 16% | — |
Willingness to say “I don’t know” collapsed from 44% to 3%. Accuracy dropped from 27% to 9% — roughly one third of the baseline. Confidence rose from 30% to 76%.
Participants who would have answered correctly without AI asked the model, received a wrong answer, adopted it, and reported higher certainty than when they were working alone.
Monetary incentives were introduced in a second condition. The willingness to admit ignorance recovered to 8%. Accuracy reached 16%. Both numbers remain well below the no-AI baseline. The effect is partially suppressible with financial stakes but does not reverse.
The Cognitive Mechanism
Lead researcher Valerio Capraro described the finding at its core: “People became much worse, the accuracy was only one third, but they were twice as confident.” The mechanism being measured is what Capraro calls the capacity to say “I don’t know” — a metacognitive habit that represents the recognition of the limits of one’s own knowledge.
The study joins a line of converging research. Wharton researchers earlier this year coined the term “cognitive surrender” to describe participants accepting incorrect AI answers 80% of the time while reporting higher confidence than those working unaided. The Milano-Bicocca experiment is a tighter version of the same result: the model quality is controlled, the questions are calibrated to expose AI failure, and the effect is still substantial.
AI products are designed to answer. They are not designed to produce “I don’t know.” The humans using them are learning the same behaviour.
Structural Risk
Capraro names children as the sharpest concern. They are adopting AI tools before the metacognitive habit — knowing what you don’t know — has formed. That habit, once suppressed by a consistently confident AI interface, is harder to build later.
The timing of the preprint coincides with Common Sense Media calling Google’s AI search redesign an “unacceptable risk” for students. Google’s new AI Mode replaces links with confident generated summaries at the exact moment a link would have prompted a harder look. The structural design of both interventions is the same: AI that answers confidently rather than pointing toward further inquiry.
The study is a preprint; independent replication is needed. But the numbers align with Wharton’s prior findings closely enough that the direction of the effect is not in question. Access to AI, even wrong AI, raises confidence and suppresses the admission of ignorance. The incentive to close that gap with better calibration exists almost nowhere in current product design.