SWE-bench Test: Swapping Agent Harnesses Boosts Coding Scores More Than New Models
_lewtun · x · 2026-08-07
A developer tested 10 mainstream coding agent harnesses (e.g., Codex, Claude Code) on SWE-bench Pro across different models. The findings reveal that the choice of harness significantly impacts final scores, often more than upgrading the model itself.
- Massive score variance: Swapping harnesses boosted pass@1 from 23% to 52% on GLM-5.2, and from 15% to 36% on Gemma 4 26B-A4B.
- Rankings don't transfer: Harness performance is highly model-dependent. Vendor-specific harnesses (Codex, Claude Code) rank high with large models but drop sharply with smaller ones, while model-agnostic harnesses (crush, opencode) climb. The rank correlation between the two models' leaderboards is just -0.05.
This suggests the AI coding field may be over-indexing on tuning model weights while underestimating the engineering potential of the surrounding agent harness.
Related event: SWE-bench Tests Show Switching Agent Frameworks Beats Swapping Models(2 posts)→
More from coding & agent
- Opus 5.5 + Three.js demo stuns with water physics, fish and blossoms — nikola_mr64990 · 2026-10-16
- Grok-built linear regression bot claims $508K PnL in 125 days on Polymarket — Roger_M_Taylor · 2026-10-16
- Karpathy's knowledge engine workflow: Claude Code turns raw notes into a self-updating wiki — nikola_mr64990 · 2026-10-16
- Sierra and Meta unveil Personal Agent Protocol with Zapier as launch partner — sujingshen · 2026-10-16
- ArcKit adds 102 AI-narrated short films for commands across 14 community overlays — mcraddock · 2026-10-16
- Personal agents live or die on chaining tasks, not single-task ability — sujingshen · 2026-10-16