Stanford CS336: The Ultimate Free LLM Engineering Course
techNmak · x · 2026-08-17
Stanford's CS336 (Language Modeling from Scratch) for Spring 2026 is highlighted as one of the best free playlists for LLM engineering, featuring 19 lectures that cover almost the entire LLM stack.
Course Scope:
- Build the Model: Tokenization, Transformers, architectures, MoE
- Understand Hardware: FLOPs, memory, GPUs, TPUs
- Optimization: Triton, GPU kernels, parallelism, distributed training
- Training: Scaling laws, data collection, filtering, deduplication
- Deployment: Inference, evaluation
- Post-training: SFT, RLHF, RLVR
- Multimodality
Hands-on Focus:
Unlike most AI classes, this course requires students to write a significant amount of code (an order of magnitude more than average). Key projects include:
- Implementing tokenizers, Transformers, and optimizers from scratch
- Writing FlashAttention2 in Triton
- Building memory-efficient distributed training pipelines
- Processing raw Common Crawl dumps into pretraining data
- Fitting scaling laws
- Training a math reasoning model using SFT and reinforcement learning
The course is freely available on YouTube and is highly recommended for those transitioning from building with LLMs to understanding how they are built.
More from Research
- Reproducible low-level artifacts found in ChatGPT image generation — DickHorner · 2026-08-17
- Best Paper Awards at RLC Workshop: LLMs as Policy Optimizers and Robust Multi-Task RL — EmmaBrunskill · 2026-08-17
- How to Use LLMs to Extract Structured Register Mappings from Unseen Industrial Manuals? Reddit User Seeks Architecture Advice — Plenty_Shine_8250 · 2026-08-17
- Opinion: Pretraining is an uncontrollable Shoggoth unlike RL — brianryhuang · 2026-08-17
- Study Reveals 'Hostile Takeover' by Immune Cells in Aging Brain — EricTopol · 2026-08-17
- If Recall Is the Bottleneck for Parametric Factuality, What Does It Mean for SciCode? — geoffwolfe · 2026-08-17