Test Shows Agent Harness Choice Can Cause Up to 30x Difference in Token Consumption
机器之心 · wechat · 2026-07-31
Composio tested three agent frameworks (Claude Code, Hermes, and Kimi Code) using the same model (Kimi K3). Across 28 identical tasks, success rates were similar, but token consumption varied by up to 30x. The median token usage for Claude Code was 340k, nearly 6 times that of Kimi Code (61k), bringing the average cost per task to $2 compared to $0.22.
Sebastian Raschka noted that this inefficiency stems from the input side: Claude's harness repeatedly feeds massive historical context across multi-turn interactions. Furthermore, controlled experiments by Writer confirmed that optimizing the orchestration layer alone—without changing models—reduced costs by 41% and cut latency by 44%. This proves that in the era of commoditized models, the 'harness tax' is the primary determinant of an agent's operational cost and efficiency.
Related event: Composio Test: Agent Frameworks Show 30x Token Gap(7 posts)→
More from coding & agent
- 'LLM code is sloppy' rebutted with real ffmpeg code: identical if/else branches in a top library — kingslyj · 2026-10-16
- Agent actions need checks outside the model: NVIDIA demos lean on guardrails amid wave of real agent incidents — Creamy-And-Crowded · 2026-10-16
- Vitalik builds "paranoid mode" agent wrapper with Tor, zkAPI and local prompt rewriting — DavideCrapis · 2026-10-16
- Agent fine-tunes Qwen3.5-0.8B with synthetic data: 4.7%→76.3% accuracy, 11.9× faster — ben_burtenshaw · 2026-10-16
- Beam-501B agent demo: solo beats 5-agent team on ProgramBench rebuild task — ben_burtenshaw · 2026-10-16
- Beam agent builds BeamSpeak, a local Parakeet dictation app for macOS — ben_burtenshaw · 2026-10-16