- AI hallucinations are mathematically inevitable, not engineering bugs.
- Benchmarks reward guessing over admitting uncertainty.
- Advanced reasoning models hallucinate more on factual questions.
Adam Tauman Kalai asked a language model for his own birthday. It gave three different answers, all wrong.
Kalai, an OpenAI researcher, wasn't surprised. He and his colleagues had just finished proving mathematically why such errors are unavoidable. Their September 2025 paper demonstrates that AI hallucinations aren't engineering failures waiting to be patched. They are statistical certainties baked into how language models learn.
What is an AI hallucination?
An AI hallucination occurs when a language model generates a statement that sounds plausible but is factually incorrect. Unlike a database lookup error, the model doesn't "know" it's wrong. It produces the most statistically probable next words, whether or not they happen to be true.
Trained to Guess, Rewarded for Bluffing
The paper, co-authored with Ofir Nachum, Edwin Zhang, and Georgia Tech's Santosh Vempala, traces the problem to two sources.
First, language models learn by predicting the next word in a sequence. This process accumulates small errors that compound. Even with perfect training data, the generative error rate is at least twice the misclassification rate for simple true-or-false judgments.
Second, the evaluation systems that rank AI models actively reward confident guessing. The team analyzed ten major benchmarks, including GPQA, MMLU-Pro, and SWE-bench. Nine used binary grading: correct answers score one point, everything else scores zero.
Under those rules, admitting uncertainty guarantees the same score as being completely wrong.
AI hallucinations are a necessary evil of generative AI. (Science Reader).
The math is unforgiving. Whatever the odds of guessing correctly, the expected score from guessing always exceeds the score from abstaining under binary grading.
Models learn to bluff because bluffing pays.
The Fix That Would Break the Product
OpenAI acknowledged the problem openly. The company noted that GPT-5 hallucinates less often, especially when reasoning, but the errors persist. More tellingly, their own advanced reasoning models performed worse on factual questions. The o1 model hallucinated 16% of the time on person-specific queries. The newer o3 and o4-mini models reached 33% and 48%.
Smarter models, it seems, are more creative guessers.
The proposed remedy involves teaching models to assess their own confidence before responding, then penalizing wrong guesses more heavily than honest uncertainty. The mathematics work. The business case does not.
Fixing hallucinations would kill the product.
Wei Xing, University of Sheffield
If ChatGPT began saying "I don't know" to 30% of queries, users would leave. Xing, who analyzed the paper for The Conversation, drew a parallel from his air-quality monitoring work: displays showing confident but inaccurate readings attract more engagement than systems that honestly flag uncertainty.
People prefer a confident wrong answer to an honest shrug.
Scoreboards Shape Behavior
The paper's most practical argument targets the benchmark system itself.
More On AI And LLM
AI Bias: How Language Models Amplify What They Copy
Researchers studying AI bias thought they were building digital twins. What they created instead were caricatures.
→Rather than building new hallucination-specific tests, the authors call for reforming the existing leaderboards that dominate AI development. Penalize incorrect guesses. Award partial credit for admitting ignorance. The same logic that standardized tests have used for decades with negative marking.
Update: By early 2026, the trajectory had become clearer. The best-performing models reduced hallucination rates dramatically on summarization tasks, but reasoning-intensive models pushed error rates in the opposite direction. GPTZero found over 50 papers with AI-hallucinated citations at ICLR 2026, each having passed multiple rounds of peer review.
The scoreboards, it appears, haven't changed fast enough.
Sources
- Primary Source: Why Language Models Hallucinate (arXiv, Kalai et al., 2025)
- Additional Context:
- Why language models hallucinate (OpenAI)
- Why OpenAI's solution to AI hallucinations would kill ChatGPT tomorrow (The Conversation, Wei Xing)
- AI hallucinates because it's trained to fake answers it doesn't know (Science/AAAS)
Fact Check: Claim-by-Claim Verification Verified
The article accurately summarizes the key claims, authors, and findings from the primary arXiv paper by OpenAI researchers, with appropriate hedging on implications.
Commentary
- Article attributes "9 out of 10 benchmarks" directly; paper's meta-analysis supports this for top benchmarks, though exact count is from secondary sources like Science.org.
- Hallucination rates for o1/o3/o4-mini (16%, 33%, 48%) not in primary arXiv paper but align with OpenAI's reported trends in SimpleQA eval on their blog.
- Update on ICLR 2026 lacks direct verification here but fits paper's prediction of persistent issues without benchmark changes.
Sources used for verification
Academic/Peer-reviewed:
- Why Language Models Hallucinate - arXiv
Other reliable sources:
- Why language models hallucinate - OpenAI
- Why OpenAI's solution to AI hallucinations would kill ChatGPT tomorrow - theconversation.com
- AI hallucinates because it's trained to fake answers it doesn't know - science.org
Fact-checked by Perplexity Sonar Pro on 2026-03-09
