Sebastian Raschka's deep dive: how RoPE, NoPE and hybrids encode position in modern LLMs
rasbt · x · 2026-11-15
Sebastian Raschka published a comprehensive article on how LLMs encode token positions, covering classic and contemporary methods with visual examples, math, and a PyTorch implementation.
- Gallery stats: 27 of 32 architectures in 2025 used RoPE-only, but of 21 entries from Aug 26 to Nov 5, 2026, only 12 are RoPE-only while NoPE and hybrid designs account for 9;
- Explains why relative schemes work: rotating Q and K means the dot product depends only on the angle difference (26 × cos 60° = 13);
- Details partial RoPE (rotating a subset of dimensions) and hybrid tricks: SmolLM3 skips RoPE in every fourth block, Inkling adds a learned bias to attention scores;
- Written as a standalone building block ahead of his upcoming LLM architecture trends analysis.
More from Models
- Sebastian Raschka publishes from-scratch RoPE tutorial and survey of positional embeddings in modern LLMs — rasbt · 2026-11-15
- Claude wrote and trained a language model small enough to fit on a floppy disk — Background-Wafer-548 · 2026-11-15
- Forecast Chart Projects GPT-7.5 by Oct 2027 and Claude 7 by Apr 2028 — flowersslop · 2026-11-15
- Codex Pro users petition OpenAI to let unused weekly usage roll over — InTeGeR78 · 2026-11-15
- Rumored Gemini 4 Krypton spotted in Arena grey tests, nailing 20 art styles in one prompt — gaganghotra_ · 2026-11-15
- "Phonebook-trained models can't write Shakespeare": data distribution beats any harness — gerardsans · 2026-11-15