176-Setting Study Dissects Coding Agent Harnesses: Context Management Matters Most on Tight Budgets
_akhaliq · x · 2026-09-19
A new paper, "An Empirical Study of Harness Design for Coding Agents," builds a modular harness that keeps the execution loop fixed while varying planning, tool interfaces, and context management, evaluated across 176 settings with four models on SWE-Bench Verified and Terminal-Bench 2.1.
Key findings:
- Context management matters most with tight context budgets: it prevents overflow from killing runs early; benefits shrink as windows grow.
- Staging elision before LLM summarization is the most efficient strategy, controlling peak context with similar success rates.
- Planning shifts from accuracy scaffold to efficiency aid: for weak models it keeps trajectories alive; for strong models it mainly cuts redundant post-edit verification.
- Predefined tools help bash-weak models; bash-only interfaces cut cost for bash-capable ones.
Related event: Zoom's Empirical Study Dissects Coding Agent Harness Design(7 posts)→
More from coding & agent
- Free course covers the 4 hard problems of production generative media agents — Al_Grigor · 2026-11-04
- Quota reset night: Codex back to 100% while Claude shows just 10% left — rudrank · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Building a 7-GPU 132GB VRAM local inference rig: consolidate on EPYC or split across two systems? — infectiousstupidity · 2026-11-04
- Founder of 50-person, 12-country team says he talks to AI agents more than humans — Scobleizer · 2026-11-04
- Pi launches a minimal agent harness; Zammoth adds AI engineer to Mac music app — solyarisoftware · 2026-11-04