Harness optimization doubles Qwen solutions on SWE-bench
rohanpaul_ai · x · 2026-08-31
This paper studies how changing the harness affects fixed-model task results. It compares keeping full history vs. compressing old outputs/handling stalls. Under 20k token limits, the optimized config boosted Qwen2.5-Coder's complete solutions on SWE-bench Verified from 43 to 72, significantly improving F2PF metrics. This proves the critical role of engineering architecture in agent performance.
Related event: Harness Optimization Doubles Qwen's SWE-bench Performance(2 posts)→
More from coding & agent
- Simon Willison: code mode is the future of tool calling, if it can be sandboxed — springrod · 2026-10-16
- Agent named 'Ilya' reportedly scores 1.1pt higher on benchmarks — daniel_mac8 · 2026-10-16
- Gemini CLI fix: substring matching inlined unrequested binary files into model context — priyanshu1976 · 2026-10-16
- Building AI agents on real healthcare claims data: the domain work that makes them work — alex_verem · 2026-10-16
- OpenAI stealth-testing mysterious internal model 'Aeon' in Codex desktop app and Dots — cedric_chee · 2026-10-16
- Vibe-coded iPhone Duo sniping bot built with Codex in 17 minutes, 41 tests passing — xiaohu · 2026-10-16