Karpathy's 'overfit first, regularize later' still rules large-scale post-training
rdesh26 · x · 2026-09-05
Karpathy's classic A Recipe for Training Neural Networks methodology still holds in the large-scale post-training era. A reader of a new long-form writeup highlighted its practical details:
- Experimentation methodology: run ablations on a 35B MoE before scaling to 397B
- The overfitting run: verify the model can actually learn a small task set before burning compute on a full RL run — a practice popularized by Lewtun that saves enormous time
- TITO implementation: building the harness around /completions instead of /chat/completions
The core principle — "overfit first, regularize later" — spans the whole evolution from the CNN era to RL post-training.
More from coding & agent
- Astra implements entire design system in Figma, taking over designers' grunt work — AIandDesign · 2026-09-05
- Runway New Hire: Figma Optional, Real Work Done in Cursor, First Product Ready to Ship — zielinskiwoj · 2026-09-05
- LoRA Creator Edward Hu Publishes Guide on Post-Training Open-Source Models with RL — iamrobotbear · 2026-09-05
- GPT-6 Astra has unique blind spots, so this dev routes coding to it and reviews elsewhere — PawelHuryn · 2026-09-05
- Bot Mesh launches a social network where AI agents get identities, pages and pay each other — Daniel_Farinax · 2026-09-05
- Heads-up: you must update Codex CLI to access GPT-6 Astra — BLUECOW009 · 2026-09-05