Free University-Level Course Dives Into the Math of Transformers and Token Dynamics
CatAstro_Piyush · x · 2026-11-19
A free university-level resource, "Transformers: Attention and Token Dynamics", explores the mathematics behind Transformer architectures.
It covers self-attention (queries, keys, values, and the kernel view), the 1/√d softmax scaling, multi-head attention, positional encoding, Transformer blocks, and masked language modeling, plus token dynamics with mathematical proofs, a clustering theorem, low-temperature softmax, and modern large-scale attention. The material builds the architecture from the equations in 100 lines, trains a miniature BERT on Shakespeare, then strips attention down to a hardmax interacting-particle system where a clustering theorem is proved. Includes Python code; suited to readers with some math background.
More from Research
- tla-kit: the full TLA+ toolchain in one reproducible container, plus a 14-tool MCP server — Scottie-2222 · 2026-11-19
- Anshul Kundaje: research artifacts should be version-controlled, not frozen as a single 'final version' — anshulkundaje · 2026-11-19
- META-Robot from HKUST completes 44/56 real-robot trials with self-repair — jiqizhixin · 2026-11-19
- Sebastian Raschka Adds Appendix on Positional Embeddings, Including YaRN and ALiBi — rasbt · 2026-11-19
- Ant Lingbo embodied AI contest: 15 teams, one shared VLA model, 48-hour real-robot showdown — 机器之心 · 2026-11-19
- STOPD cuts on-policy distillation rollout tokens by 25–48% with unbiased random truncation, 2.2x speedup — 机器之心 · 2026-11-19