Harness experiment: cheaper model + guardrails beats frontier model at 33% less cost
blaizedsouza · x · 2026-10-14
Reshared by Martin Fowler: Naresh Jain compared coding agents building the same API service with and without a behavioral harness.
- Frontier GPT-6 Sol passed all 10 runs either way, but with the harness it used 10.5x tokens, took 3x longer, and cost 7x more.
- Cost-optimized GPT-6 Luna succeeded only 3/10 unharnessed but 10/10 with the harness, at $0.273 per run — 33% cheaper than Sol's $0.406.
Takeaway: a behavioral harness (architectural contracts + deterministic test feedback in the inner loop, evaluation harness in the outer loop) may be a practical way to get dependable results from smaller, cost-optimized models.
More from coding & agent
- Is moving messy code into a function real refactoring, or just boxing? — jimmykoppel · 2026-10-14
- Claude Code 2.1.299 is about to ship, says tracker account — ClaudeCodeLog · 2026-10-14
- Agents-first software emerges as a trend as Notion and others ship MCPs — Historical_Kick3793 · 2026-10-14
- Steering coding agents in large repos: missing guardrails produce more slop — yoyoyoav · 2026-10-14
- Coinbase fine-tuned Qwen3.5-9B for fraud prevention — beat frontier models, cheaper — lmoroney · 2026-10-14
- Founder asks: what are you actually letting your AI agents buy? — kevinfee · 2026-10-14