Generator outruns verifier: 250 samples lift DeepSeek-Coder to 56% on SWE-bench
le_james94 · x · 2026-09-16
The author spent three weeks working through Stanford Online's CS 329A lectures (taught by Azalia Mirhoseini and Aakanksha Chowdhery), and identifies one argument running through the whole course: the generator has raced far ahead of the verifier.
He backs it with data: DeepSeek-Coder-V2-Instruct resolves only 15.9% of SWE-bench Lite issues with 1 sample, but 56% with 250 samples—possible only because a test suite tells you which sample was correct. In other words, the pile of generated answers keeps getting better, while your ability to reach into it and pick the right one does not.
The thread then gives a section-by-section recap of the lectures, all anchored on this generator–verifier gap.
Related event: Stanford Opens CS329A on Self-Improving AI Agents(5 posts)→
More from coding & agent
- Same Prompt, Four Models Behind One MCP: Only One Got It Right — rohanpaul_ai · 2026-09-16
- Shumer: Skip Complex Setups, One Manager Agent Session Is Enough — mattshumer_ · 2026-09-16
- Google Cloud API Gateway now acts as a remote MCP server for existing REST APIs — rseroter · 2026-09-16
- Dev ports Blender 5.1 MCP add-on to Blender 3.6, AI builds a full owl scene from one prompt — SpinachOk9137 · 2026-09-16
- Lyft cut support agent ship time from 6 months to 1-2 weeks with LangGraph and LangSmith — LangChain · 2026-09-16
- Text-only AI agent beats Doom at ~10 calls/sec, costing about $7 per hour — hackgoofer · 2026-09-16