Cheap model's overnight bug-fix PRs: only 6 of 14 survived a stronger model's review
Aggressive-Narwhal-3 · reddit · 2026-10-03
An experiment on the author's own repos: during an API discount window, a cheap DeepSeek model ground through audit-found bugs overnight.
- Round 1: 14 PRs, all re-reviewed by GPT-6.1 Sol; 6 merged after CI, 8 closed. Failures included a fix that broke other valid inputs, tests needing tools absent from CI, a bypass that still worked, a key test never run in CI, and a new test that couldn't catch the bug even when code was deliberately broken.
- Round 2 flipped roles: a stronger model designs each fix, another reviews the design, and the cheap model only implements. So far 7 PRs, none approved yet.
- The author asks the community where cheap-model fixes fail and what plan/implement/review split actually works.
More from coding & agent
- Devs flag GPT's defensive programming for hiding bugs; OpenAI asks for /feedback cases — pvncher · 2026-11-19
- Flutter ships A2UI 1.0: agents respond with real interactive widgets, not chat text — rseroter · 2026-11-19
- Skills vs MCP vs RAG vs Memory: the four context mechanisms AI agents actually need — MaryamMiradi · 2026-11-19
- Anthropic details agent memory: 'dreaming' lets Claude organize its experience — daniel_mac8 · 2026-11-19
- The case for externalized memory as the real fix for continual learning — daniel_mac8 · 2026-11-19
- FT: US tech outages blamed on rush to fix bugs found by AI — nordicinst · 2026-11-19