Coding agent cost breakdown: Claude Code tops Kotlin benchmark at 85.7% but burns 3.6x fewer tokens than Codex
Evening_Hawk_7470 · reddit · 2026-10-24
The author compiled public benchmark data from SWE-rebench, DeepSWE v1.1, and the 105-task JetBrains Kotlin benchmark across six leading coding agents (Oct 2026), arguing that token pricing on a model card says little about the real cost of a merged PR.
Key numbers:
- Claude Code: 85.71% (90/105) on the Kotlin benchmark at 10.59M avg tokens (Opus 4.7 xhigh) — best solve rate, but locked to Anthropic models with no free tier.
- JetBrains Junie: 61.6% Pass@1 on SWE-rebench at $0.25/task; 81.90% on Kotlin but 19.98M tokens; 10+ hosted models with zero-markup BYOK, free Qwen cloud/local tiers, and plan approval via persistent .junie/plans Markdown specs.
- OpenAI Codex: 81.90% on Kotlin but 38.78M avg tokens — 2x Junie and 3.6x Claude Code; open-source CLI, --oss local runtime.
- Cursor: 79.05% at 25.56M tokens, multi-model subscription plus Bugbot inline PR autofix.
Workflow takeaways: a cheaper model running three test-verification loops can burn 2x the tokens yet still finish at a fraction of the cost; letting a frontier model edit 15 files before anyone checks its assumptions often ends in a reverted branch. Citing the 2026 JetBrains Developer Ecosystem Survey: developers rewrite 34% of agent-generated code on average, and 42% say reviewing an AI PR takes more effort than reviewing a colleague's — a net +12pt supervision tax that rises to +17pt for senior engineers.
More from coding & agent
- Agent orchestration tip: skip goal, use a manager agent with scheduled polling — pvncher · 2026-10-24
- Programming isn't dead — it just got harder, says dev sharing Claude terminal workflow — MoonL88537 · 2026-10-24
- Earendil Working to Steer Agent Skills Toward an Open, Collaborative Future — dsp_ · 2026-10-24
- Edera: threat modeling AI agents with STRIDE — and why containment is futile — One_Paint8721 · 2026-10-24
- A ~3K-token AGENTS.md rule set to stop coding agents from going off the rails — akumaburn · 2026-10-24
- Directionally Local launches private coding-agent infra that runs on edge hardware you own — sahilypatel · 2026-10-24