CMU and JetBrains study: doubling AI token spend yields only ~10% more commits
rohanpaul_ai · x · 2026-11-12
A Carnegie Mellon and JetBrains paper, "The AI Era's Lines of Code?", examines whether token spend measures engineering productivity.
- Data: 5 weeks of coding-agent token usage linked to commits for 404 JetBrains engineers, plus a survey of 412 employees; technical staff used agents beyond coding in nearly 9 of 10 evaluations.
- Key findings: within the same engineer, doubling token spend was associated with only 10% more commits; among weeks with similar commit counts, the 90th-percentile week spent 39–240x the tokens of the 10th-percentile week.
- Weak explanatory power: available work metrics explained only 38% of between-engineer and 13% of week-to-week spend variation.
- Takeaway: token spend measures how much engineers use agents, not output or value — use it for budgeting only, and evaluate agent-assisted work via task and outcome evidence.
Companies already use token spend for budgets, productivity judgment, and even ranking top users — this study directly challenges that practice.
Related event: CMU Study: Doubling Token Spend Yields Only ~10% More Commits(2 posts)→
More from coding & agent
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- Dev's favorite AI use case: Claude monitors competitors and emails alerts — doooyle · 2026-11-12
- LLM controls vocal synth to perform Ave Maria in 4-part harmony — AAAzzam · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Claude watches competitors around the clock and emails me what matters — GlenBradley · 2026-11-12
- Developers say Gemini 4 Carbon is notably better at favicon design and micro-interactions — AI_Andrew · 2026-11-12