Cosma Shalizi's notebooks read Attention and Transformers as kernel smoothing and Markov models
tak3sh8 · x · 2026-10-13
CMU statistician Cosma Shalizi's public notebooks on "Attention", "Transformers", and LLMs are recirculating — written, he admits, partly out of jealousy and fear of technical obsolescence.
- On attention: he argues attention is essentially kernel smoothing, critiques multi-head naming, discusses identification failures, and revives McDermott's critique of "wishful mnemonics" for AI curricula.
- On language models: he explores "isn't this just a Markov model?", LLMs vs Lempel-Ziv compression, next-token vs longer-range prediction, hunches about prompt leakage, and Gopnik-style "libraries".
- Caveat: the author stresses these are personal working notes, not intellectual authority, and likely contain errors — yet their contrarian statistical lens has proven influential (nattyover wrote a column prompted by them).
A valuable read for anyone wanting an information-theoretic perspective on Transformer limitations.
More from Research
- Google DeepMind hiring Summer 2027 research intern for continual learning — TheGradient · 2026-10-13
- Researcher argues reward-free imitation of verified completions inherits most of RLVR's problems — QuintinPope5 · 2026-10-13
- Proposal: replace online RL with imitation on verified successful trajectories — QuintinPope5 · 2026-10-13
- Thinking Inertia: LLMs keep reasoning even when explicitly told not to, Stanford-led study finds — StanfordUniversity · 2026-10-13
- SJTU's MAVLN: first systematic multi-agent vision-language navigation benchmark with 11,724 episodes — SJTU · 2026-10-13
- MSAR multi-stream attention residuals beat prior variants at 18B/28B MoE, 1.58x iso-loss speedup — SJTU · 2026-10-13