Cross-Entropy and LLM Training Explained
3Blue1Brown · youtube · 2026-07-16
This video explains why cross-entropy serves as the loss function for training language models through the lens of information theory and compression. It reviews the intuition behind optimal coding and language trees, defines cross-entropy, and illustrates how it measures the cost of using one distribution to describe another.
The latter half connects these concepts to LLM pre-training, explaining why minimizing cross-entropy is equivalent to improving next-token prediction, and discusses why it is the optimal loss choice. The video also touches upon distillation and KL divergence, weaving these concepts into a cohesive narrative.
More from Research
- Structural ensembles beat single predictions in TCR:pMHC generalization study — quaidmorris · 2026-07-22
- Structural ensembles, not single predictions, drive robust TCR:pMHC generalization — quaidmorris · 2026-07-22
- enFoldX turns AlphaFold3 ensemble noise into a TCR–peptide–MHC predictor — quaidmorris · 2026-07-22
- RSS launches under OMSF to push structural biology data modeling at scale — MoAlQuraishi · 2026-07-22
- enFoldX tops 8 neoantigen scans and an unseen-peptide benchmark — quaidmorris · 2026-07-22
- enFoldX reaches AUC 0.82 on human VDJdb and transfers to mouse at 0.76 — quaidmorris · 2026-07-22