Benchmark of 21 model-harness combos: agent framework changes cost, not success rate
A viral study benchmarked 7 models across 3 coding harnesses (21 combos), finding that the choice of agent framework barely affects success rates but dramatically changes cost.
2026-09-19 ~ 2026-09-20 · 2 related posts
- Episode 1: 7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success(2026-09-17, 7 posts)
- Episode 2: Benchmark of 21 model-harness combos: agent framework changes cost, not success rate(2026-09-19, 2 posts)
- Episode 3: Berkeley's Harness Tax Study: Harness Choice Barely Affects Success Rate but Multiplies Cost(2026-09-21, 5 posts)
- 21 model-harness pairs tested: framework choice barely moves success rate but swings cost — iScienceLuvr · 2026-09-19
- Evaluating 7 Models Across Claude Code, Codex, and Pi: Harness Choice Drives Cost, Not Success — CShorten30 · 2026-09-20