Free University-Level Course Dives Into the Math of Transformers and Token Dynamics

CatAstro_Piyush · x · 2026-11-19

A free university-level resource, "Transformers: Attention and Token Dynamics", explores the mathematics behind Transformer architectures.

It covers self-attention (queries, keys, values, and the kernel view), the 1/√d softmax scaling, multi-head attention, positional encoding, Transformer blocks, and masked language modeling, plus token dynamics with mathematical proofs, a clustering theorem, low-temperature softmax, and modern large-scale attention. The material builds the architecture from the equations in 100 lines, trains a miniature BERT on Shakespeare, then strips attention down to a hardmax interacting-particle system where a clustering theorem is proved. Includes Python code; suited to readers with some math background.

Original post →

More from Research

Research channel →