Agent Harness Matters More Than the Model: SWE-bench Scores Swing by 10+ Points
Future_AGI · reddit · 2026-07-22
The author points out that current AI Agent discussions overly focus on the model itself (e.g., GPT vs. Claude), neglecting the core factor determining an agent's practical performance: the model's harness.
The harness refers to the wrapping code that turns a model into an agent, including:
- The loop deciding when to call tools and when to stop
- How tool outputs are fed back into the context
- Memory trimming strategies when the context window fills
- Error handling and retry mechanisms
Key Takeaways:
- Leaderboard scores are model-plus-harness numbers: The same model can perform vastly differently under varying harnesses. For instance, on SWE-bench Pro, Claude Opus 4.5 scores 46% under one standardized scaffold and 55% under another. Merely changing the scaffold can cause 10 to 20-point swings.
- Debug unreliability by inspecting the harness first: When an agent is flaky, blindly switching models is often ineffective. The root cause usually lies in truncated contexts, tool errors failing to propagate back into the loop, or bad states stacked by retries. Cleaning up these harness-level issues improves reliability much more than swapping models.
More from coding & agent
- Tencent Hunyuan’s Hy3 ranks #25 in Agent Arena and #2 among open models in Frontend Code Arena — arena · 2026-07-22
- Bindu Reddy says OpenAI and Anthropic already rely heavily on vibe coding — bindureddy · 2026-07-22
- Karpathy says Codex and Claude Code need user control over subagent roles and model choice — emollick · 2026-07-22
- Writing too few CLAUDE.md files keeps forcing models to rebuild context — TheZachMueller · 2026-07-22
- Telemetry can’t prove an AI intrusion was fully autonomous — cyb3rops · 2026-07-22
- Helmholtz Munich and TUM open fully funded PhD spots in trustworthy AI for science — zeynepakata · 2026-07-22