Agent Harness Matters More Than the Model: SWE-bench Scores Swing by 10+ Points
Future_AGI · reddit · 2026-07-22
The author points out that current AI Agent discussions overly focus on the model itself (e.g., GPT vs. Claude), neglecting the core factor determining an agent's practical performance: the model's harness.
The harness refers to the wrapping code that turns a model into an agent, including:
- The loop deciding when to call tools and when to stop
- How tool outputs are fed back into the context
- Memory trimming strategies when the context window fills
- Error handling and retry mechanisms
Key Takeaways:
- Leaderboard scores are model-plus-harness numbers: The same model can perform vastly differently under varying harnesses. For instance, on SWE-bench Pro, Claude Opus 4.5 scores 46% under one standardized scaffold and 55% under another. Merely changing the scaffold can cause 10 to 20-point swings.
- Debug unreliability by inspecting the harness first: When an agent is flaky, blindly switching models is often ineffective. The root cause usually lies in truncated contexts, tool errors failing to propagate back into the loop, or bad states stacked by retries. Cleaning up these harness-level issues improves reliability much more than swapping models.
More from coding & agent
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11
- How Do You Catch Behavioral Regressions in LLM Agents Between Releases? — Beautiful_Belt_601 · 2026-09-11
- Treating agents like 50 First Dates: a 3-layer context system so every conversation doesn't start from zero — evielync · 2026-09-11
- Running the Firefox MCP on Android via Termux, ngrok, and mcp-proxy — Nervous-Strain7544 · 2026-09-11
- Run Firefox MCP on Android: Termux + ngrok tunnel tutorial — Nervous-Strain7544 · 2026-09-11