Build an LLM Training Stack from Scratch
Y0oshi2 · reddit · 2026-07-13
Frustrated by the "magic" of abstraction layers like transformers, trl, and peft, the author built a training stack entirely from pure PyTorch base operators to make the entire training process transparent and readable.
Implementations
- Native implementations of Pre-training, SFT (with prompt masking), DPO, and GRPO/RLVR
- Decoder-only Transformer featuring RoPE, RMSNorm, SwiGLU, GQA, and KV cache
- Custom byte-level BPE tokenizer
- Memory-mapped datasets to minimize memory footprint
- A unified codebase compatible with CUDA, Apple Silicon (MPS), and CPU
Verifiability
- Configurations provided for training a 17M parameter model
- Capable of running the full pipeline locally on TinyStories to quickly generate coherent text
- Includes an interactive CLI that automatically assembles and executes training commands
The author encourages the community to review the code, suggest optimizations, or use it as a learning resource to grasp underlying mechanisms like GRPO and KV cache.
More from Research
- OmniSearch puts text, images, audio, and video into one semantic search space — victorialslocum · 2026-07-21
- Cold Spring Harbor Asia sets a genome biology conference in Suzhou for Oct. 12–16 — jmuiuc · 2026-07-21
- A clean counterexample shows a map can be locally diffeomorphic yet globally fold — Algomancer · 2026-07-21
- Xiaohongshu’s dots-note-3.0 gets a perfect IMO score and becomes the world’s second gold model — 量子位 · 2026-07-21
- Statistical theory paper studies how fast signatures learn in path regression — chaumian · 2026-07-21
- PROWL uses a world model to keep Minecraft agents exploring after failures — nathanbenaich · 2026-07-21