LLM Code Fixing Gauntlet: Qwen3.6 Wins, Retry Mechanism Key
sysadmin420 · reddit · 2026-08-30
The author built a rigorous local LLM evaluation arena using 6 real historical bugs from their repo to test code fixing capabilities (must pass regression tests without breaking the full suite).
Results (3 retries):
- qwen3.6:35b-a3b: 3/6
- devstral:24b: 2/6
- Qwen3-coder:30b: 1/6
- qwen2.5-coder:14b: 1/6
- deepseek-coder-v2:16b: 0/6
Key Findings:
- Introducing a retry loop (feeding errors back) significantly changed results, showing models are better iterators than one-shot solvers.
- In vision tests, qwen3-vl:8b perfectly read numbers from rendered images while llava:13b hallucinated heavily.
Public results/videos: https://informant.reiners.io/gauntlet
More from coding & agent
- Agent workflows break in prod despite passing sandbox tests: how to test? — Common_Dream9420 · 2026-08-30
- TrustScoreAgent: Open reputation registry for agent API calls — TrustScoreAgent · 2026-08-30
- BrainAPI beats Mem0/Zep; bottleneck is infra, not models — shbong · 2026-08-30
- Arcturus Bot Integrates GPT-image-2, Showcasing Image Gen in Group Chats — MikePFrank · 2026-08-30
- Adapting ColBERT MaxSim to 10,000-D Bipolar HDC for Edge Memory Gating — Equivalent-Flan-1590 · 2026-08-30
- Varying definitions of Agent memory architecture by tech leaders — bibryam · 2026-08-30