Writing Code vs. Shipping Code: The AI Productivity Gap

If you had to summarise the AI coding boom in one number, most companies would pick the biggest one. “240% more commits.” That figure is real. It is also the least useful number available.

A 2026 NBER working paper — Writing Code vs. Shipping Code by Mert Demirer, Leon Musolff and Liyuan Yang (Working Paper 35275) — used telemetry from more than 500,000 GitHub developers to follow the effect of AI tools through the entire production chain. The finding reframes how you should measure any AI investment you have made.

The attenuation is the story

Software development moves through six stages: writing lines of code, organizing files, committing, submitting pull requests, merging, and releasing. The researchers tracked each one.

Three generations of AI tooling were measured:

Tool generation	Example	Cumulative effect on commits
Autocomplete	in-editor suggestions	+30%
Sync agents	Claude Code, working alongside you	+180%
Async agents	OpenAI Codex, running remotely	+240%

Now watch the same 240% decay as it moves downstream:

+240%  commits
  +80%  projects
  +30%  actual releases

Only an eighth of the commit-level gain survives all the way to releases. Seven-eighths of it evaporates before anything reaches a user.

They confirmed this across four major app marketplaces: adoption produced a moderate increase in the number of new apps, but no increase in total usage. More software shipped. Not more software used.

Why: the weak link

The authors call it the weak-link hypothesis. AI is highly productive at specific tasks, but aggregate output is capped by complementary tasks still performed by humans whose productivity did not change.

Their estimate of the elasticity of substitution between AI and human effort is 0.23 — meaning AI effort replaces only about a quarter of the human effort it displaces. The two are complements, not substitutes. Every line of code AI writes still needs someone to decide whether it is correct.

This is the single most important number in the paper. It is why “we added AI” and “we are shipping faster” are two different claims.

Where the pressure actually showed up

Corroborating research makes the mechanism concrete — but the two most-cited datasets here come from different reports, and reading them as one story produces numbers that cannot all be true at once.

Faros AI’s 2025 AI Productivity Paradox report (10,000+ developers, 1,255 teams, analysed as of June 2025) found teams using AI heavily completed 21% more tasks and merged 98% more pull requests — while PR review time leapt by 91%, and organisational delivery metrics stayed flat.

Faros AI’s 2026 Acceleration Whiplash report is a separate dataset — 22,000 developers across 4,000 teams, March 2026 data — and its numbers differ sharply: tasks up 33.7%, merge rate up just 16.2% rather than 98%, and median review time up 441.5% rather than 91%. Addy Osmani analysed this second dataset for O’Reilly Radar in June 2026:

Metric	Change
Code churn	+861%
Incidents per PR	+242.7%
Bugs per developer	up 54% (vs 9% in the 2025 dataset)
Median review duration	+441.5%
PRs merged with zero review	+31.3%

The bugs-per-developer row is a comparison between two separate datasets, not a rise inside one period — and Faros frames it that way itself.

Good process did not protect these teams. The volume simply outran it.

CodeRabbit separately studied 470 open-source pull requests from December 2025 — 320 AI-coauthored against 150 inferred human-only — and found AI changes carried roughly 1.7× more issues. Worth noting the study’s own limit: “human-only” was inferred from the absence of AI co-author signals, not verified.

GitClear’s cohort research (2,172 developer-weeks, Sept–Dec 2025) puts the output-versus-value gap most bluntly. Heavy daily AI users author somewhere between 4× and 10× more work than non-users. The productivity gain that survives scrutiny is far smaller — and disputed even by the authors: GitClear reports median Diff Delta up 12% for regular AI users, while GitKraken’s co-published report on the same dataset puts regular users at 25%. GitClear’s CEO also concedes part of even that residual is selection bias, with strong performers concentrating in AI cohorts.

The Stack Overflow Developer Survey 2025 shows the same split between individual and organisational gains. About 69% of agent users agree agents have increased their productivity, and roughly 70% agree agents cut the time spent on specific tasks. But only 17% agree agents improved collaboration within their team — Stack Overflow itself calls it the lowest-rated impact by a wide margin. Meanwhile 66% report being frustrated by AI solutions that are “almost right, but not quite”, and 45.2% say debugging AI-generated code is more time-consuming.

The part nobody costed

There is a second cost, and it is not a number. A July 2026 preprint, (Im)Paired Programming, ran 54 students building a website either with a coding agent or with a chatbot that only emits short generic snippets — 59 recruited, 5 dropped for failing attention checks.

Agents improved initial task completion. They hurt code comprehension, low-effort interaction patterns — copy-and-paste prompting, auto-accepted edits — correlated with worse understanding, and the agents did not improve the follow-up extension task. What made it worse: participants still preferred the agents, because they were fast and easy.

Faster output, weaker understanding, lower oversight. That is the direction of travel if nobody changes the workflow.

What to actually measure

Stop counting commits. Track:

1. Release throughput — not commits. Shipped work per cycle.
2. Review burden — median time to first review, and time in review queue.
3. Rework rate — what fraction of merged work gets reopened or reverted.
4. Defect rate per change — the bug-per-developer gap between Faros’s 2025 and 2026 datasets is the number that predicts your incident page.
5. Time to understanding — can a human still reason about the code after the agent leaves?

The honest version

AI coding tools are a real lever. They are not the lever you thought they were. They compress the cheapest stage of development and leave the expensive stages untouched — then add load to them.

The bottleneck was never typing speed. It was deciding whether the work was correct. That step is still human, still slow, and still the only one that determines whether you have a product or a repository.

📖 Related: The Toggle Tax covers the same attenuation pattern outside software.

Sources: Demirer, Musolff & Yang, Writing Code vs. Shipping Code (NBER Working Paper 35275, rev. September 2026) · Faros AI, The AI Productivity Paradox Report 2025 · Faros AI, comparing 2025 and 2026 telemetry · MIT Sloan summary · O’Reilly Radar, Agentic Code Review (Addy Osmani, June 2026) · Stack Overflow Developer Survey 2025 · (Im)Paired Programming (arXiv:2607.26375, July 2026)