Claude Code sends 16,581 tokens per step yet scores equal a 100-line mini-harness
bendee983 · x · 2026-10-25
Two separate studies show that harness complexity doesn't necessarily improve coding agent accuracy—and can triple your token bill. Per VentureBeat, Claude Code sends 16,581 tokens of fixed context at every step, while the 100-line mini-SWE-agent sends just 829—yet their SWE-bench Verified scores are statistically equivalent. The author warns that vendor self-reported benchmarks deserve skepticism: run-to-run variance on the same model-harness config can exceed the gains vendors attribute to their harnesses. Build your own evals and make your own judgment.
More from coding & agent
- print3d: open-source agent skill turns photos or text into Bambu Lab 3D prints — udmrzn · 2026-10-25
- Dev launches gaming-focused remote MCP server for squad-finding, seeking testers — Rashad8705 · 2026-10-25
- Redditor compares the agent boom: Claude Code, OpenClaw, Hermes vs hosted rivals — Odd-Aside456 · 2026-10-25
- Jev ecosystem explodes: 5 open-source apps from a 7.1s flight-search agent to SQL-in-English — udmrzn · 2026-10-25
- PhD student says Claude now does most of his thesis work, down to the draft — GarrisonLovely · 2026-10-25
- Google's Jon Barron shares his full AI paper-writing workflow with Gemini — lmoroney · 2026-10-25