Tests Show Massive Token Cost Gap Among AI Agent Frameworks

The Composio team recently conducted cost and performance tests on various LLM and Agent framework combinations, revealing a major cost trap in AI coding: despite similar task success rates, Token consumption varies drastically across frameworks, with differences of up to 30x.

Confirmed

Similar Success Rates: Composio ran the same model (Kimi K3) on 28 identical tasks using Claude Code, Hermes, and Kimi Code. The results showed closely matched success rates, with Kimi Code succeeding in 22 tasks, Hermes in 21, and Claude Code in 20.
Drastic Token Consumption Gap: Despite similar success rates, Token usage varied enormously. The median task consumption for Claude Code was around 340,000 Tokens, compared to just 61,000 for Kimi Code—a 6x difference. Independent testing by developer rasbt confirmed this, finding that Claude Code consumes 2-3x more Tokens than other frameworks.
Speed Differences: Beyond Token efficiency, execution speeds also varied significantly across different frameworks.

Why it matters

Harnesses are Cost Black Holes: Tests indicate that choosing different Agent frameworks directly leads to massive disparities in API costs, sometimes by tens of multiples. This means that in AI coding applications, the overhead of the framework's "wrapper" has become a critical cost center that cannot be ignored. Developers must factor in Token consumption efficiency as a key criterion during framework selection.

2026-07-29 ~ 2026-07-31 · 5 related posts

Primary sources

Agent Harness Matters: Wrong Framework Can Cost 30x More Tokens — aigclink · 2026-07-30 12:47 (UTC+08:00)
Composio benchmark finds similar success rates but wide speed gaps across coding harnesses — Teknium · 2026-07-29 22:01 (UTC+08:00)
Test Shows Claude Code Consumes 6x More Tokens Per Task Than Kimi Code — zainhas · 2026-07-30 01:13 (UTC+08:00)

[source] Composio benchmark finds similar success rates but wide speed gaps across coding harnesses — Teknium · 2026-07-29
[source] Test Shows Claude Code Consumes 6x More Tokens Per Task Than Kimi Code — zainhas · 2026-07-30
Testing Kimi K3: Up to 30x Token Cost Difference Across Agent Harnesses — evijit · 2026-07-30
[source] Agent Harness Matters: Wrong Framework Can Cost 30x More Tokens — aigclink · 2026-07-30
Claude Code Burns 2-3x More Tokens Than Other Agent Harnesses — RexDouglass · 2026-07-31