Agent Harness Matters: Wrong Framework Can Cost 30x More Tokens
aigclink · x · 2026-07-30
Composio recently conducted tests on the cost and performance of combining LLMs with different agent harnesses, revealing several core trends in the AI coding landscape:
- Harness is a Cost Trap: When running Kimi K3 on 28 identical tasks, Claude Code, Hermes, and Kimi Code achieved similar success rates, but token consumption varied by up to 30x. Choosing the wrong shell can inflate the bill by dozens of times.
- Open-Source Catches Up: On a 14-task agentic benchmark, the open-source Kimi K3 matched the closed-source Claude Sonnet 5 exactly (9 passes, 5 failures), with differences only in cost.
- Closed-Source Internal Competition: On 23 tasks, GPT-5.6 Sol beat Opus 5 (22/23 vs 20/23 pass rate) while being faster and cheaper.
As top models converge on the ability to simply "get the job done," competition is shifting back to business fundamentals: token spend, time, and overall cost.
Related event: Composio Test: Agent Frameworks Show 30x Token Gap(7 posts)→
More from coding & agent
- Connected steps are not a workflow: nobody designs the handoff — usehive · 2026-10-16
- Making Blender animations with Opus, Astra and Argon: how you prompt still matters — chongdashu · 2026-10-16
- Running PolyStrata + Cline coding agent on a $0-fancy laptop at ~90 tok/s — rpwoerk · 2026-10-16
- How do you attribute agent gains: model training or infrastructure? — jrxna · 2026-10-16
- Feed Claude your books, blogs and top engineers' code for claude.md — 'it now writes better code than me' — blaizedsouza · 2026-10-16
- Open-source /label plugin auto-names and colors Claude Code sessions in one step — hermanschutte · 2026-10-16