Composio Test: Agent Frameworks Show 30x Token Gap

Composio recently released a cost and performance test of LLM and agent framework combinations, revealing a cost trap in AI coding: with similar task success rates, token consumption and monetary costs vary dramatically across frameworks, up to 30x. This shows that the 'wrapper' overhead of the framework itself has become a significant cost core.

Confirmed

Similar success rates: Composio ran the same model (Kimi K3) on 28 identical tasks across different frameworks. Success rates were close: Kimi Code 22/28, Hermes 21/28, Claude Code 20/28.
Huge token consumption gap: Despite similar success, token usage varied greatly. Claude Code's median task consumed 340k tokens vs Kimi Code's 61k, a 6x difference; overall max gap reached 30x. Developer rasbt independently confirmed Claude Code uses 2-3x more tokens than other frameworks.
Per-task cost differences: Using model pricing, average per-task costs varied significantly. Hermes Agent and Pi Agent were cheapest at $0.39 and $0.40, while Claude Code averaged $1.47, nearly 4x higher.
Speed differences: Besides token efficiency, execution speed also varied greatly.

Why it matters

Harness is a cost black hole: The test shows that choosing different agent frameworks can lead to tens of times difference in API costs. This means the 'wrapper' overhead is now a critical cost factor in AI coding, and developers must consider token efficiency when selecting frameworks.

2026-07-29 ~ 2026-07-31 · 7 related posts

Full story(4 episodes)→

Episode 1: Composio Test: Agent Frameworks Show 30x Token Gap(2026-07-29, 7 posts)
Episode 2: Agent Framework Benchmarks: Kimi K3 Shines in Lightweight Efficiency(2026-07-31, 2 posts)
Episode 3: Kimi K3 Matches Opus in Coding, Tops Open-Source(2026-08-01, 2 posts)
Episode 4: Coding Agent Cost Comparison: Claude Code is 3.7x Pricier Than Open Source(2026-08-01, 3 posts)

Primary sources

Testing Kimi K3: Up to 30x Token Cost Difference Across Agent Harnesses — evijit · 2026-07-30 02:33 (UTC+08:00)
Agent Task Cost Benchmark: Claude Code Costs Nearly 4x More Than Hermes — Teknium · 2026-07-31 20:46 (UTC+08:00)
Claude Code Burns 2-3x More Tokens Than Other Agent Harnesses — RexDouglass · 2026-07-31 02:08 (UTC+08:00)

Composio benchmark finds similar success rates but wide speed gaps across coding harnesses — Teknium · 2026-07-29
Test Shows Claude Code Consumes 6x More Tokens Per Task Than Kimi Code — zainhas · 2026-07-30
[source] Testing Kimi K3: Up to 30x Token Cost Difference Across Agent Harnesses — evijit · 2026-07-30
Agent Harness Matters: Wrong Framework Can Cost 30x More Tokens — aigclink · 2026-07-30
[source] Claude Code Burns 2-3x More Tokens Than Other Agent Harnesses — RexDouglass · 2026-07-31
[source] Agent Task Cost Benchmark: Claude Code Costs Nearly 4x More Than Hermes — Teknium · 2026-07-31

1 near-duplicate retellings: 机器之心