Local quants vs Claude on 9 real private bugs: 3-bit costs thinking time, not score
FeydRowan · reddit · 2026-11-02
A Redditor benchmarked his local models against Claude on 9 real, never-public C++/CUDA regression bugs from his own inference engine (capy-flash, 2x RTX 5070 Ti, 96 GB RAM). The agent runs sandboxed with no network or compiler, reasoning from code only; patches are scored by a hidden key and a blind Claude Opus 5 judge (r = 0.79 correlation), 3 runs per task.
Results (judge score 0-1):
- Claude Opus 5.5: 0.74, just 16k tokens / 3 min per task — clearly ahead
- Claude Sonnet 5.5: 0.58 (8k tokens / 1 min)
- Qwen3.8-Flash-Next NVFP4: 0.50, best local, at Haiku 5.5 level but needing 53k tokens / 10 min
- Qwen3.8-27B 3-bit: 0.39; Gemma 4 26B-A4B only 0.07
Key quantization finding:
- 3-bit on Flash-Next's experts only: score unchanged (-0.02), effectively free
- 3-bit on the 27B: score slightly up (+0.05, within noise) but 40% more tokens per task (x1.40), pushing per-bug time from 13 to 18 minutes
- Conclusion: quantization's cost shows up as longer thinking to reach the same answer — judging by tok/s alone misleads on value per bug fixed
Caveats: ±0.2 confidence intervals; models within 0.1 are a coin flip.
More from coding & agent
- Open-source insect sticker library ships with agent-run generate-and-review pipeline — AlchainHust · 2026-11-02
- 19-year-old claims $750K profit from Claude Code-built trading bot started with $68 — aftahi_ai · 2026-11-02
- Solo dev vibe-codes a superhero MMO with Opus 5.5 in 6 weeks, burning 3B tokens — superpikapoo · 2026-11-02
- Genshin-like action game demo built in 24 hours with Opus 5.5 and Tripo 3D — AlchainHust · 2026-11-02
- Combining Codex, agent-desktop and browser-use cut a 3-hour accounting job to under an hour — EXM7777 · 2026-11-02
- Dev ships a survivors-like extraction shooter in 10 days with Claude Code: 535 commits, no image generators — spiritosito · 2026-11-02