Reading Stanford CS329A in Depth: Generators Have Outpaced Verifiers

James Le spent 3 weeks working through Stanford Online's Fall 2025 course CS329A, "Self-Improving AI Agents," publicly available on YouTube. Taught by Azalia Mirhoseini (ex-Google) and Aakanksha Chowdhery across nine lectures, he wrote a long-form recap for each one. What makes this series of notes worth attention: it systematizes the core tension in today's AI capability landscape—the gap between generators and verifiers.

Confirmed

• The course's central thesis is that generator capabilities are now far ahead of verifiers, a thread running through all nine lectures.
• Using the example that sampling DeepSeek-Coder 250 times solves 56% of problems, it illustrates how large volumes of generated samples can exceed single-shot verification ability—the generation-verification asymmetry is becoming increasingly pronounced.
• DeepSeekMath-V2's approach turns verification from a constraint into a product: train the verifier first, then keep scaling verification compute as the generator improves so verification always stays a step ahead; but whether this holds beyond mathematics, the author admits no one knows.
• AI capability measurement is splitting into two tracks: METR evaluations show o3's 50% time horizon is roughly equivalent to 110 minutes of human work, doubling about every 7 months; meanwhile, in DeepScholar-style literature review benchmarks, no system scores above 31%.

Why it matters

• If generation keeps outpacing verification, research focus will shift to "how to scale verification compute"—verifiers could become the new competitive battleground and product form.
• The coexistence of the doubling time-horizon narrative for long-horizon tasks with low scores on concrete tasks (like literature reviews) shows a single benchmark no longer suffices to describe AI capability; the evaluation landscape itself is fragmenting.

2026-09-16 ~ 2026-09-16 · 5 related posts

• [source] Stanford CS329A publishes 9-lecture course on self-improving AI agents — le_james94 · 2026-09-16
• Generator outruns verifier: 250 samples lift DeepSeek-Coder to 56% on SWE-bench — le_james94 · 2026-09-16
• [source] o3's time horizon hits ~110 minutes yet no system clears 31% on DeepScholar-Bench — le_james94 · 2026-09-16
• DeepSeekMath-V2 makes verification the product, scaling verifier compute ahead of the generator — le_james94 · 2026-09-16
• [source] 3 weeks through Stanford CS329A: the generator has outrun the verifier — le_james94 · 2026-09-16