Baseten's 83B-token eval finds coding agent cost varies 4.4x across harnesses
baseten · x · 2026-11-18
Baseten partnered with Runta to build FrontierHarness Eval v1.1, spending 83B tokens across 27,280 agent runs to compare 22 harnesses on 5 open models.
Key findings:
- Cost per pass can vary by up to 4.4x across harnesses on the same model;
- Codex CLI ranks 1st on cost with GLM-5.3 but only 10th of 22 with Kimi-K3 — when you switch models, consider switching harnesses too;
- Only Pi never left the top 6 on cost across all models.
Advice for teams running coding agents on open models: maintain your own evals and re-test your harness on every new model release.
More from coding & agent
- Dev builds 1:1 scale Singapore island with Astra: 1,000+ subagents, 100+ hours — gabrielchua · 2026-11-18
- SilentGuard: harness-level invariant checks cut agent API silent-failure errors from 74% to 1.7% — Genentech · 2026-11-18
- GPT-6.2 Astra and Sol arrive: hands-on visual coding battle vs Claude Fable 5.2 — arena · 2026-11-18
- Developer shows off Runway MCP: call video generation right from coding agents — tlakomy · 2026-11-18
- Early tester builds playable 16v16 WWII browser game Omaha Front with GPT-6.2 Sol — VraserX · 2026-11-18
- LangChain's Chase declares proactive agents the next wave, shares build lessons — BraceSproul · 2026-11-18