Generator outruns verifier: 250 samples lift DeepSeek-Coder to 56% on SWE-bench
le_james94 · x · 2026-09-16
The author spent three weeks working through Stanford Online's CS 329A lectures (taught by Azalia Mirhoseini and Aakanksha Chowdhery), and identifies one argument running through the whole course: the generator has raced far ahead of the verifier.
He backs it with data: DeepSeek-Coder-V2-Instruct resolves only 15.9% of SWE-bench Lite issues with 1 sample, but 56% with 250 samples—possible only because a test suite tells you which sample was correct. In other words, the pile of generated answers keeps getting better, while your ability to reach into it and pick the right one does not.
The thread then gives a section-by-section recap of the lectures, all anchored on this generator–verifier gap.
Related event: Three Weeks with Stanford CS329A: Generators Have Outrun Verifiers(9 posts)→
More from coding & agent
- GitHub Actions' ubuntu-slim runner is lighter than ubuntu-latest, but caps jobs at 15 minutes — DanielLockyer · 2026-11-02
- GitHub Actions has a lesser-known ubuntu-slim runner: 1 CPU, 5GB RAM, cheaper CI — DanielLockyer · 2026-11-02
- Before automating a workflow with AI, ask if anyone understands it end to end — djgoel123 · 2026-11-02
- One stale date silently broke a WhatsApp reply agent: 19 messages unanswered — petrusenko_max · 2026-11-02
- The 10X engineer is now just a SKILLS.md: 6 workflow rules for AI coding agents — mdancho84 · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02