LLM Code Fixing Gauntlet: Qwen3.6 Wins, Retry Mechanism Key
sysadmin420 · reddit · 2026-08-30
The author built a rigorous local LLM evaluation arena using 6 real historical bugs from their repo to test code fixing capabilities (must pass regression tests without breaking the full suite).
Results (3 retries):
- qwen3.6:35b-a3b: 3/6
- devstral:24b: 2/6
- Qwen3-coder:30b: 1/6
- qwen2.5-coder:14b: 1/6
- deepseek-coder-v2:16b: 0/6
Key Findings:
- Introducing a retry loop (feeding errors back) significantly changed results, showing models are better iterators than one-shot solvers.
- In vision tests, qwen3-vl:8b perfectly read numbers from rendered images while llava:13b hallucinated heavily.
Public results/videos: https://informant.reiners.io/gauntlet
More from coding & agent
- Connected steps are not a workflow: nobody designs the handoff — usehive · 2026-10-16
- Making Blender animations with Opus, Astra and Argon: how you prompt still matters — chongdashu · 2026-10-16
- Running PolyStrata + Cline coding agent on a $0-fancy laptop at ~90 tok/s — rpwoerk · 2026-10-16
- How do you attribute agent gains: model training or infrastructure? — jrxna · 2026-10-16
- Feed Claude your books, blogs and top engineers' code for claude.md — 'it now writes better code than me' — blaizedsouza · 2026-10-16
- Open-source /label plugin auto-names and colors Claude Code sessions in one step — hermanschutte · 2026-10-16