Agent Harness Matters More Than the Model: SWE-bench Scores Swing by 10+ Points

Future_AGI · reddit · 2026-07-22

The author points out that current AI Agent discussions overly focus on the model itself (e.g., GPT vs. Claude), neglecting the core factor determining an agent's practical performance: the model's harness.

The harness refers to the wrapping code that turns a model into an agent, including:

The loop deciding when to call tools and when to stop
How tool outputs are fed back into the context
Memory trimming strategies when the context window fills
Error handling and retry mechanisms

Key Takeaways:

Leaderboard scores are model-plus-harness numbers: The same model can perform vastly differently under varying harnesses. For instance, on SWE-bench Pro, Claude Opus 4.5 scores 46% under one standardized scaffold and 55% under another. Merely changing the scaffold can cause 10 to 20-point swings.
Debug unreliability by inspecting the harness first: When an agent is flaky, blindly switching models is often ineffective. The root cause usually lies in truncated contexts, tool errors failing to propagate back into the loop, or bad states stacked by retries. Cleaning up these harness-level issues improves reliability much more than swapping models.

Original post →

Agent Harness Matters More Than the Model: SWE-bench Scores Swing by 10+ Points

More from coding & agent