9.9K-star GitHub repo trains an LLM from scratch in plain PyTorch, pretraining to GRPO
techNmak · x · 2026-09-20
The 9.9K-star GitHub repo train-llm-from-scratch packs an entire LLM engineering course into one codebase.
- Builds the Transformer from raw text in plain PyTorch based on Attention is All You Need, with no transformers, trl or peft hiding the training loops
- Covers the full journey: pretraining → SFT → reward modeling → DPO → PPO → GRPO, ending at an aligned, reasoning-style model
- The smallest model is only 13M parameters, so everything runs on a single GPU and you can see what changes at each stage
- Ships with notebooks, configs, docs and a Streamlit UI for line-by-line learning
More from Research
- XGEN Labs unveils generative world simulation JING+DAO, tops WBench leaderboard — hey_abusiddik · 2026-09-20
- Schmidhuber: LLMs aren't truly creative because they lack compression progress — SchmidhuberAI · 2026-09-20
- Jev tested on 8,054 NASA Kepler signals: 54.2% accuracy, loses to a simple 3-rule baseline — This_Cell_1829 · 2026-09-20
- François Fleuret nicknames his training curves; researchers admit they curse baselines too — giffmana · 2026-09-20
- HN: How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip — petrusenko_max · 2026-09-20
- FlashNorm: two lines of algebra buy 33-35% speedup — and a CUDA race that made the model echo the past — AI Engineer · 2026-09-20