LLM Code Fixing Gauntlet: Qwen3.6 Wins, Retry Mechanism Key
sysadmin420 · reddit · 2026-08-30
The author built a rigorous local LLM evaluation arena using 6 real historical bugs from their repo to test code fixing capabilities (must pass regression tests without breaking the full suite).
Results (3 retries):
- qwen3.6:35b-a3b: 3/6
- devstral:24b: 2/6
- Qwen3-coder:30b: 1/6
- qwen2.5-coder:14b: 1/6
- deepseek-coder-v2:16b: 0/6
Key Findings:
- Introducing a retry loop (feeding errors back) significantly changed results, showing models are better iterators than one-shot solvers.
- In vision tests, qwen3-vl:8b perfectly read numbers from rendered images while llava:13b hallucinated heavily.
Public results/videos: https://informant.reiners.io/gauntlet
More from coding & agent
- Simon Willison: code mode is the future of tool calling, if it can be sandboxed — springrod · 2026-10-16
- Dev workflow: Astra for tedious tasks, Claude for ideas not implementation — menhguin · 2026-10-16
- Agent named 'Ilya' reportedly scores 1.1pt higher on benchmarks — daniel_mac8 · 2026-10-16
- Gemini CLI fix: substring matching inlined unrequested binary files into model context — priyanshu1976 · 2026-10-16
- Building AI agents on real healthcare claims data: the domain work that makes them work — alex_verem · 2026-10-16
- OpenAI stealth-testing mysterious internal model 'Aeon' in Codex desktop app and Dots — cedric_chee · 2026-10-16