SWE-bench Pro Shows Harness Swings Scores More Than Model Upgrades
OfirPress · x · 2026-08-09
Recent NeurIPS reviews and SWE-bench Pro evaluations reveal that AI coding agents are often over-engineered with complex harnesses.
- Harness > Model: Swapping the agent harness can drastically shift performance, moving pass@1 from 23% to 52% for GLM-5.2 on SWE-bench Pro—a wider gap than most model upgrades provide.
- No Universal Winner: There is no one-size-fits-all harness. A heavily optimized harness for large models (like Codex) can rank near the bottom for smaller models. Rank correlation across models is virtually zero.
- Simplicity Wins: Researchers argue that a minimalist harness (e.g., just providing a bash environment and getting out of the way) is actually the strongest approach. If a model genuinely needs a new tool, it can synthesize one on the fly (e.g., live-SWE-agent).
Related event: SWE-bench Tests Show Switching Agent Frameworks Beats Swapping Models(2 posts)→
More from coding & agent
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- Claude Code and Codex now autocomplete the user's next action — tokenbender · 2026-10-16
- Hermes Agent ships official Reverse Engineer Anything plugin with one-command setup — Teknium · 2026-10-16
- MLExpert: 35B MoE Agent Trained to Autonomously Handle Machine Learning Engineering — Shangheng Du · 2026-10-16
- Cowork-Gym: Alibaba Synthesizes 64.9K Agent Work Environments, Boosting Qwen by 8.7 Points — Anni Zou · 2026-10-16
- MMSearch-OPD: Joint RL and On-Policy Distillation for Long-Horizon Multimodal Search Agents — PekingUniversity · 2026-10-16