Harnesses make a small coding model match frontier reliability at 33% lower cost
bibryam · x · 2026-10-16
A ThoughtWorks experiment compared coding agents building the same API service with and without a behavioral harness (inner behavioral loop plus an outer evaluation loop with 178 automated acceptance tests).
- Frontier model GPT-6 Sol: passed 10/10 runs without a harness. Adding one cost 7x more, 10.5x tokens, 3x runtime — no gains.
- Cost-optimized GPT-6 Luna: only 3/10 unharnessed, but 10/10 with the harness, at 33% less per run than Sol ($0.273 vs $0.406).
The author frames the harness as "executable attention": architectural contracts, API specs, and deterministic test feedback turn an unreliable small model into a dependable one. Takeaway: more harness isn't always better — the right setup depends on model and task, and harness + small model can beat a frontier model on cost-adjusted reliability.
More from coding & agent
- Engineers Should Mine Gong Calls for Ideas That Never Became Tickets — curious_vii · 2026-10-16
- One 80-Word Prompt, Overnight AI Agent Built a Site With 205 Bakeries Mapped — jamesbrooksco · 2026-10-16
- Visual Lean: drag-and-drop GUI for the Natural Numbers Game runs Lean 4 in WASM — MilesCranmer · 2026-10-16
- Fine-Tuned 0.6B Router Beats Its 27B Sibling: Small Specialized Models Win — mshubham · 2026-10-16
- Engineering Leader: AI Got Me Coding Again — I Built Our Feature-Flag System With an Agent — rseroter · 2026-10-16
- Dev has lead agent update t3 code profile so threads survive rate limits — NERDDISCO · 2026-10-16