Same model bills 5x more in a different harness: 21 combos tested across 60 tasks
CShorten30 · x · 2026-09-22
HarnessTax tested 21 model-harness combos across 60 tasks and found cost moved far more than success rate did: $1.33 vs $0.67 per attempt for the same model, while success ranged only from 96.7% to 97.8%. Nine of twelve lean setups beat vendor defaults. Claude Code and Pi averaged 15 turns per task, but heavy setups carried over 10x the initial context since longer instructions and larger tool definitions ride along on every call. Co-author melissapan notes harness choices run on tribal knowledge, making it unclear which setup wins on cost for your workload.
More from coding & agent
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Ten wild Gemini 4 Carbon demos in 24 hours: CUDA meadows, V8 engines, mech stomps — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Hamel Husain's complete AI evals framework for PMs and engineers — aakashgupta · 2026-11-12
- AI coding in one year: from unusable to full automation with no code review — angelaqdai · 2026-11-12
- Interactions Tool Searches Web Animations by Motion for Claude Code to Rebuild — yuwen_lu_ · 2026-11-12