Zoom's 176-setting ablation study reveals which coding harness components actually matter
dair_ai · x · 2026-09-19
Zoom researchers ran a systematic ablation of coding harnesses: with the execution loop held fixed, they varied planning, action space, and context management one at a time across 176 matched settings, four models, and both SWE-Bench Verified and Terminal-Bench 2.1.
Key findings:
- Context management pays off more as the context window tightens, and its benefit comes mostly from preventing overflow failures rather than better reasoning
- Staging rule-based elision before LLM summarization gave the best accuracy-to-cost ratio of five strategies tested
- Making elided content recoverable added machinery models rarely used, with no accuracy gain
- Planning changed role with model strength: it lifted success rates for the weakest model, but barely moved accuracy for stronger ones
Directly useful for anyone maintaining a hand-built coding harness.
Related event: Zoom's Empirical Study Dissects Coding Agent Harness Design(7 posts)→
More from coding & agent
- Musk cites unsustainable Grok Bot costs as Hermes offers one-command agent migration — Teknium · 2026-11-04
- Open-source agent Cindy launches: cross-device tasks, files and model sharing — yangyi · 2026-11-04
- Multi-Agent Loop Workflow Curates 1,132 Top Bilibili Videos Into a Website — huangyun_122 · 2026-11-04
- Graphic designer builds NASA-data 3D Solar System in 2 months with Claude Code, no coding — Shevei1342 · 2026-11-04
- How to Run a 5-Person Office With Claude: Assistant, Bookkeeper, PM, Analyst & Office Manager — evielync · 2026-11-04
- Replit's User Ops lead built an AI app to prioritize and route support tickets — amasad · 2026-11-04