Harness design, not the model, drives coding agent scores: 176-setup study quantifies it
alex_verem · x · 2026-09-25
Researchers from UMass Amherst, Zoom, Emory, and UNC Charlotte held models constant and varied only the harness—planning, action space, context management—across 176 matched setups on SWE-Bench Verified and Terminal-Bench 2.1.
Key findings:
- Context management matters most when the context window is tight, mostly by preventing overflow failures. Rule-based elision staged before LLM summarization was most efficient; making elided content recoverable added machinery models rarely used.
- Planning is an accuracy scaffold for weak models (Nemotron-3 30B dropped from 25.2% to 13.6% without a plan, stalling after a median of 5 turns) but a cost saver for strong models—about 30% cheaper with under 2 points accuracy change.
- Tools split the same way: Mistral Medium 3.5 fell from 68.6% to 45.4% without predefined file tools, while Nemotron-3 550B solved more tasks with a bare terminal at 53% lower cost.
- With a 32k window and no memory management, 78.7% of GitHub tasks failed from overflow; trimming/summarizing lifted the same model to 51–58%.
Conclusion: no single best setup—pick components per model, task, and budget. A benchmark score measures model plus harness, and the harness controls a large share of it.
More from coding & agent
- Obsidian Starter Kit v4 turns your vault into a typed AI knowledge base with MCP and 375 skills — dSebastien · 2026-09-25
- RouterGrowth launches as 'OpenRouter for sales' — one agent prompt found 20 leads for $0.72 — aliscodes · 2026-09-25
- Dev says Claude remotely filed a lawsuit, answered emails and pitched his project with a demo video — aloncarmel · 2026-09-25
- Who's responsible when your AI agent spawns another agent? Multi-agent authority chains lack an audit trail — Fantastic-Sleep-3352 · 2026-09-25
- Developer says Codex has fully replaced Claude Code as his primary coding tool — eptwts · 2026-09-25
- Weco preprint: agent self-improves its own software 7 times over 8 days, gains transfer to 4 benchmarks — VraserX · 2026-09-25