Open-source 12-chapter guide builds a modern LLM from scratch with every line commented
tom_doerr · x · 2026-10-16
The open-source repo how-to-train-your-gpt (3.7k stars) is a 12-chapter, 7,500+ line interactive textbook for building and training a modern LLama 3-style LLM from scratch. Every line — tokenizer, embeddings, attention, training loop, inference engine — is hand-written and annotated, backed by 28 standalone explainers covering RoPE, RMSNorm, SwiGLU, KV cache, AdamW and mixed precision, plus step-by-step narrative walkthroughs aimed at readers with zero ML background.
More from Research
- SWE-Duel: self-scaling adversarial arena has coding agents hide and catch bugs — Consistent_Fox_2990 · 2026-10-16
- Removing context conditioning makes image conditioning stronger in flow-guided video generation — pixlpa · 2026-10-16
- Researcher slams papers whose stacked changes beat even weak baselines barely — retr0jirachi · 2026-10-16
- FrankenSim builds a physics-based virtual orchestra, piano and drum demos live — doodlestein · 2026-10-16
- FoveaVLA replaces wrist cameras with learned foveation, hitting 70.8% success — Haoyu_Xiong_ · 2026-10-16
- ApolloPFN: a zero-shot forecasting network that natively handles exogenous covariates — burkov · 2026-10-16