3 weeks through Stanford CS329A: the generator has outrun the verifier
le_james94 · x · 2026-09-16
James Le spent 3 weeks working through all 9 lectures of Stanford's CS329A "Self-Improving AI Agents" (Azalia Mirhoseini, Aakanksha Chowdhery) and wrote a section-by-section recap.
- The through-line: the generator has raced ahead of the verifier — generating candidates is far easier than verifying them, and that gap is where agent research's hard problems live.
- METR's chart shows agent task-horizon doubling every 7 months (GPT-2 through Claude 3.7 Sonnet), but at 80% reliability the horizon collapses — both lines are real.
- Self-improvement is one loop everywhere: generate candidates → verify (unit tests, reward models, LLMs, humans) → select or train on survivors → repeat.
- Lecture 9 highlights DeepSeekMath-V2: treating verification as the product, scaling verification compute to stay ahead as the generator improves — whether that holds outside math (legal reasoning, experimental design, untestable codebases) is unknown.
- Topics span test-time compute scaling, robust verification, tools/code feedback, planning, RL train-time scaling, search and deep research agents, agentic evals, and long-horizon tasks.
Related event: Three Weeks with Stanford CS329A: Generators Have Outrun Verifiers(9 posts)→
More from coding & agent
- GitHub Actions has a lesser-known ubuntu-slim runner: 1 CPU, 5GB RAM, cheaper CI — DanielLockyer · 2026-11-02
- Before automating a workflow with AI, ask if anyone understands it end to end — djgoel123 · 2026-11-02
- One stale date silently broke a WhatsApp reply agent: 19 messages unanswered — petrusenko_max · 2026-11-02
- The 10X engineer is now just a SKILLS.md: 6 workflow rules for AI coding agents — mdancho84 · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02