Zoom's 176-setting ablation study reveals which coding harness components actually matter
dair_ai · x · 2026-09-19
Zoom researchers ran a systematic ablation of coding harnesses: with the execution loop held fixed, they varied planning, action space, and context management one at a time across 176 matched settings, four models, and both SWE-Bench Verified and Terminal-Bench 2.1.
Key findings:
- Context management pays off more as the context window tightens, and its benefit comes mostly from preventing overflow failures rather than better reasoning
- Staging rule-based elision before LLM summarization gave the best accuracy-to-cost ratio of five strategies tested
- Making elided content recoverable added machinery models rarely used, with no accuracy gain
- Planning changed role with model strength: it lifted success rates for the weakest model, but barely moved accuracy for stronger ones
Directly useful for anyone maintaining a hand-built coding harness.
Related event: Zoom's 176-Experiment Study Dissects Coding Agent Harness Components(7 posts)→
More from coding & agent
- Muse Opens Connector Platform: Developers Plug In APIs to Reach Agent Users — jffwng · 2026-09-19
- Jasper's open RL guide: shaping rewards to train a search agent end-to-end — simonguozirui · 2026-09-19
- Make Your Evals Better: More Edge Cases and Real User Inputs, Not Synthetic Slop — cephaloform · 2026-09-19
- Agent runs full SEO audit of Opendoor in 4.5 minutes, surfacing 500+ link opportunities — morganb · 2026-09-19
- AWS's Marc Brooker on the future of code review: LLMs plus automated reasoning — alexisgallagher · 2026-09-19
- Thorsten Ball demos agent picking its next command from shell history — IanArawjo · 2026-09-19