SWE-bench Pro Shows Harness Swings Scores More Than Model Upgrades
OfirPress · x · 2026-08-09
Recent NeurIPS reviews and SWE-bench Pro evaluations reveal that AI coding agents are often over-engineered with complex harnesses.
- Harness > Model: Swapping the agent harness can drastically shift performance, moving pass@1 from 23% to 52% for GLM-5.2 on SWE-bench Pro—a wider gap than most model upgrades provide.
- No Universal Winner: There is no one-size-fits-all harness. A heavily optimized harness for large models (like Codex) can rank near the bottom for smaller models. Rank correlation across models is virtually zero.
- Simplicity Wins: Researchers argue that a minimalist harness (e.g., just providing a bash environment and getting out of the way) is actually the strongest approach. If a model genuinely needs a new tool, it can synthesize one on the fly (e.g., live-SWE-agent).
Related event: SWE-bench Tests Show Switching Agent Frameworks Beats Swapping Models(2 posts)→
More from coding & agent
- Securing Write Operations in MCP Servers: Four Practical Strategies — Content-Parking-621 · 2026-08-09
- AI Agent Given $1,000 Budget Earns Over $10K in Crypto Tips from Community — kleffew94 · 2026-08-09
- Letting My Agent Pay for Its Own APIs for 2 Months: The Real Pitfalls — VoidRyanZane · 2026-08-09
- Stanford Open-Sources Shepherd: Git-like Version Control for AI Agents — blaizedsouza · 2026-08-09
- AI Agent Boosts SQLite Performance by 59% in 8 Hours, Passing 1M Tests — rohanpaul_ai · 2026-08-09
- Nous Research Details Hermes Agent: Four Specialized Profiles for Collaboration — NousResearch · 2026-08-09