- AI coding tools slowed experienced developers by 19 percent.
- Developers believed they were 20 percent faster despite the slowdown.
- METR's randomised trial tested 16 developers on 246 real tasks.
Sixteen experienced software developers sat down to fix real bugs in code they knew well. Half the time, they could use AI coding tools - the other half the time, they worked alone.
The results surprised everyone, including the developers themselves.
Developers Felt Faster but Measured Slower
Joel Becker and his colleagues at METR, an AI safety research organisation, designed the study with unusual care. They recruited developers who had contributed to large open-source projects for years: codebases averaging over a million lines and 22,000 GitHub stars. These were not students learning to code.
The researchers assigned 246 real tasks from each developer's own repository. Bug fixes, feature additions, refactors; the ordinary work of maintaining serious software.
METR randomly told each developer whether to use Cursor Pro with Claude 3.5/3.7 Sonnet or to work unaided. Before starting, developers predicted AI would save them 24 percent of their time.
Key figure
19% slower
Developers using AI tools completed tasks 19% slower than working without them
Afterwards, the developers reported feeling 20 percent faster. Actual measurement told a different story: they were 19 percent slower.
Where the Time Actually Went
The perception gap is remarkably consistent. Developers genuinely believed AI accelerated their work while it was doing the opposite.
METR's data reveals where the hours went. With AI coding tools enabled, developers spent less time writing code and searching documentation. Instead, they spent more time prompting, reviewing AI output, and waiting.
Over half the AI-generated suggestions proved unusable. Even accepted suggestions needed substantial manual correction.
Why experienced developers fared worst
The study found AI was least helpful when developers already had deep familiarity with their codebase. For code they knew intimately, the overhead of prompting and reviewing outweighed any benefit from generated suggestions.
The developers, notably, were not beginners struggling with unfamiliar tools. They had been paid $150 per hour specifically because METR wanted seasoned professionals working on projects they understood thoroughly.
The Perception Gap Has a Pattern
David Gerard, a science communicator who covered the study, drew a pointed parallel from software engineering itself. "You never claim you've optimized your program without going in and measuring the program's actual performance," he observed in his analysis of the findings.
You never claim you've optimized your program without going in and measuring the program's actual performance.
David Gerard, science communicator
Software engineers have a wry saying: the first 90 percent of a project takes 90 percent of the time, and the last 10 percent takes the other 90 percent. Developers are famously poor at estimating effort. This study suggests they are equally poor at estimating how much a tool helps them.
The feeling of productivity and actual productivity can diverge sharply.
That divergence matters beyond coding. If professionals in a highly measurable field misjudge their own efficiency this badly, similar perception gaps likely exist wherever AI tools enter the workplace without rigorous measurement.
METR's Follow-Up Tells a More Complicated Story
METR has since expanded its research. A second study, beginning in August 2025, enrolled 57 developers across 143 repositories. The results grew murkier.
Original participants still showed an 18 percent slowdown (with wide confidence intervals). Newly recruited developers showed only a 4 percent slowdown. Selection bias complicated both figures: some developers refused to participate if denied AI tools, and others chose different types of tasks depending on whether AI was available.
By early 2026, METR acknowledged the landscape was shifting. Internal analysis of their own staff's Claude Code transcripts suggested time savings between 1.5x and 13x on certain tasks.
The organisation is now redesigning its methodology entirely. Short intensive experiments, observational data analysis, and developer-level randomisation are all being explored.
Measuring What AI Actually Does
The original finding still stands as a strikingly careful piece of evidence in a field saturated with anecdote.
More On AI Capabilities
AI Materials Discovery: 5 Things to Know
The gap between what AI predicts and what chemists can actually make has become the central drama of materials science.
→Most claims about AI coding productivity come from individual developers reporting their impressions; precisely the kind of self-assessment METR's study found unreliable.
Whether newer AI models and agentic tools have genuinely reversed the productivity deficit remains an open empirical question.
METR plans to find out, though their February 2026 update candidly admits they have not yet solved the measurement problem.
The tools will keep improving. The question is whether the measurement will keep pace.
Sources
- Primary Research: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker, Rush, Barnes, Rein; METR, 2025)
- Additional Context:
- arXiv preprint (Becker et al., 2025)
- We are Changing our Developer Productivity Experiment Design (METR, 2026)
- AI coders think they're 20% faster (David Gerard, Pivot to AI)
Fact Check: Claim-by-Claim Verification Verified
The article accurately reports METR's study findings, including the 19% slowdown, developer perceptions, methodology, and follow-up nuances, all matching primary sources.
Commentary
- Article appropriately notes study limitations (e.g., experienced devs on familiar codebases) and ongoing research; no overgeneralization to all devs/tools.
- Follow-up details on selection bias and pay reduction ($150/hr to $50/hr) add context without misrepresenting data.
Sources used for verification
Academic/Peer-reviewed:
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - arXiv
- arXiv PDF (full paper) - arXiv
Other reliable sources:
Fact-checked by Perplexity Sonar Pro on 2026-03-11
