- AI classification errors in healthcare are mathematically unavoidable, not just fixable bugs.
- A 500,000-student study showed even the best algorithms misclassified one in five.
- More data offers diminishing returns: up to 100x more for each 1% accuracy gain.
When an AI navigation tool routed Carlos Gershenson through a corn field, he didn't panic. He took notes. As a complexity scientist studying the limits of artificial intelligence, Gershenson knows that AI mistakes aren't always fixable glitches–sometimes they're mathematically inevitable.
That insight matters more than ever. A bill introduced in the U.S. House in early 2025 would let AI systems prescribe medications autonomously, without human doctors signing off. As Gershenson explains in The Conversation, his research on data classification suggests this might be dangerously premature.
Key figure
80%
Best algorithm accuracy on a 500,000-student classification task
The Problem Hiding in the Data
Gershenson and his team at Universidad Nacional Autónoma de México discovered something surprising while analyzing records from over 500,000 students. They wanted to predict which students would finish their degrees on time based on grades, age, gender, and socioeconomic status.
The best algorithms–including one they designed specifically for this task–achieved only 80% accuracy. One in five students was misclassified.
The reason wasn't weak algorithms. Many students were statistically identical in every measured way, yet some graduated on time while others didn't. An algorithm looking at two identical student profiles has no basis for choosing between them. It guesses.
In research published in July 2025, Gershenson's team proved that for certain datasets, perfect categorization is mathematically impossible. The data itself contains irreducible overlap.
What is irreducible overlap?
In data classification, irreducible overlap occurs when two different outcomes share identical measurable features. No algorithm can distinguish between them, because the distinguishing information simply isn't there. Adding more data of the same kind won't resolve it.
Why Dogs Explain Healthcare's AI Mistakes Problem
Think about training an AI to identify dog breeds using only age, weight, and height. Chihuahuas and Great Danes? Easy. An Alaskan malamute versus a Doberman pinscher? The measurements overlap too much. Some individuals from different breeds fall into identical ranges.
Medical diagnosis works the same way. Fever can signal a respiratory infection or a digestive problem. A cold might cause a cough, but not always. Different diseases produce identical symptoms; the same disease produces different symptoms in different people.
For some datasets – the core underpinning of many AI systems – AI will not perform better than chance.
Carlos Gershenson, Bingham University
What makes this significant is that more data won't necessarily help. Gershenson found that improving accuracy often requires exponentially more information - at up to 100 times the data for each 1% accuracy gain. And life throws curveballs that no dataset can anticipate: unexpected unemployment, family emergencies, pregnancies.
The complexity is irreducible.
The Legal Limbo of Robot Prescriptions
Alan Turing once said that a machine expected to be infallible cannot be intelligent, because learning requires making mistakes. Gershenson sees this tension play out in every AI system he studies, from traffic light coordination to tax evasion detection.
Humans make diagnostic errors too, of course. But when AI misdiagnoses a patient–and it will–who's responsible? The pharmaceutical company? Software developers? Insurance agencies? The pharmacy?
When AI misdiagnoses a patient–and it will–who's responsible?
The answer falls into legal limbo.
Gershenson argues that "centaurs", combinations of humans and machines, consistently outperform either alone. A doctor using AI to generate potential drug options based on a patient's medical history, genetics, and physiology? That's precision medicine, and researchers are already exploring it.
Related reading
AI Doctors Lie, Even When They Know They Are Wrong
What if your AI doctor agreed with you–even when you’re wrong? Scientists reveal how chatbots spread misinformation.
→But autonomous AI prescribing, without human oversight? The precautionary principle suggests we're not ready. And if errors are mathematically baked into how AI learns from overlapping data, human supervision in healthcare may always be necessary.
The navigation app that sent Gershenson through a corn field was annoying. An AI that prescribes the wrong medication could be deadly.
Fact Check: Claim-by-Claim Verification Verified
The article accurately summarizes Carlos Gershenson's research on AI classification limits and its implications for healthcare, with faithful reporting of key claims and sources.
Commentary
- Gershenson is currently Professor at Binghamton University but conducted the student research using UNAM data - the article's attribution is technically correct though could be clearer.
- The article appropriately frames the healthcare implications as concerns rather than proven facts.
Sources used for verification
Fact-checked by Perplexity Sonar Pro on 2025-12-12
