Agent Framework Tests Reveal Massive Token Cost Gaps

Episode 1 · Composio Test: Agent Frameworks Show 30x Token Gap (2026-07-29, 7 posts)

Composio recently released a cost and performance test of LLM and agent framework combinations, revealing a cost trap in AI coding: with similar task success rates, token consumption and monetary costs vary dramatically across frameworks, up to 30x. This shows that the 'wrapper' overhead of the framework itself has become a significant cost core.

Confirmed

• Similar success rates: Composio ran the same model (Kimi K3) on 28 identical tasks across different frameworks. Success rates were close: Kimi Code 22/28, Hermes 21/28, Claude Code 20/28.
• Huge token consumption gap: Despite similar success, token usage varied greatly. Claude Code's median task consumed 340k tokens vs Kimi Code's 61k, a 6x difference; overall max gap reached 30x. Developer rasbt independently confirmed Claude Code uses 2-3x more tokens than other frameworks.
• Per-task cost differences: Using model pricing, average per-task costs varied significantly. Hermes Agent and Pi Agent were cheapest at $0.39 and $0.40, while Claude Code averaged $1.47, nearly 4x higher.
• Speed differences: Besides token efficiency, execution speed also varied greatly.

Why it matters

• Harness is a cost black hole: The test shows that choosing different agent frameworks can lead to tens of times difference in API costs. This means the 'wrapper' overhead is now a critical cost factor in AI coding, and developers must consider token efficiency when selecting frameworks.

• Composio benchmark finds similar success rates but wide speed gaps across coding harnesses — Teknium · 2026-07-29
• Test Shows Claude Code Consumes 6x More Tokens Per Task Than Kimi Code — zainhas · 2026-07-30
• Testing Kimi K3: Up to 30x Token Cost Difference Across Agent Harnesses — evijit · 2026-07-30
• Agent Harness Matters: Wrong Framework Can Cost 30x More Tokens — aigclink · 2026-07-30
• Claude Code Burns 2-3x More Tokens Than Other Agent Harnesses — RexDouglass · 2026-07-31
• Test Shows Agent Harness Choice Can Cause Up to 30x Difference in Token Consumption — 机器之心 · 2026-07-31
• Agent Task Cost Benchmark: Claude Code Costs Nearly 4x More Than Hermes — Teknium · 2026-07-31

Episode 2 · Agent Framework Benchmarks: Kimi K3 Shines in Lightweight Efficiency (2026-07-31, 2 posts)

With models like Qwen and DeepSeek improving token efficiency, developers are exploring multiple agent frameworks. Recent benchmarks across six frameworks reveal that Kimi K3 offers excellent lightweight performance and efficiency, while Claude incurs higher operational costs.

• Benchmarking Kimi K3, GLM 5.2, and DeepSeek V4 Pro in Agent Workflows — Teknium · 2026-07-31
• Benchmarking Agent Harnesses: Kimi K3 Shines, Claude Code Costs 4x More — omarsar0 · 2026-08-01