Harness design, not the model, drives coding agent scores: 176-setup study quantifies it

alex_verem · x · 2026-09-25

Researchers from UMass Amherst, Zoom, Emory, and UNC Charlotte held models constant and varied only the harness—planning, action space, context management—across 176 matched setups on SWE-Bench Verified and Terminal-Bench 2.1.

Key findings:

Context management matters most when the context window is tight, mostly by preventing overflow failures. Rule-based elision staged before LLM summarization was most efficient; making elided content recoverable added machinery models rarely used.
Planning is an accuracy scaffold for weak models (Nemotron-3 30B dropped from 25.2% to 13.6% without a plan, stalling after a median of 5 turns) but a cost saver for strong models—about 30% cheaper with under 2 points accuracy change.
Tools split the same way: Mistral Medium 3.5 fell from 68.6% to 45.4% without predefined file tools, while Nemotron-3 550B solved more tasks with a bare terminal at 53% lower cost.
With a 32k window and no memory management, 78.7% of GitHub tasks failed from overflow; trimming/summarizing lifted the same model to 51–58%.

Conclusion: no single best setup—pick components per model, task, and budget. A benchmark score measures model plus harness, and the harness controls a large share of it.

Original post →

Harness design, not the model, drives coding agent scores: 176-setup study quantifies it

More from coding & agent