Sebastian Raschka publishes from-scratch RoPE tutorial and survey of positional embeddings in modern LLMs
rasbt · x · 2026-11-15
Sebastian Raschka released "Understanding Positional Embeddings in Modern LLMs," filling a gap in his LLM building-blocks series with a from-scratch RoPE walkthrough featuring visual examples, light math, and a PyTorch implementation.
- Covers classic schemes: GPT-2's learned absolute position vectors and the original Transformer's fixed sine/cosine encodings;
- Explains RoPE mechanics: rotating queries and keys inside attention so dot products depend only on relative angle;
- Surveys recent variants: NoPE, partial RoPE, and hybrids — e.g. SmolLM3 skips RoPE in every fourth block, Inkling adds a learned bias to attention scores;
- Includes stats over his LLM Architecture Gallery showing RoPE-only still dominant in 2025–2026 but NoPE and hybrid designs gaining share.
More from Models
- Claude wrote and trained a language model small enough to fit on a floppy disk — Background-Wafer-548 · 2026-11-15
- Forecast Chart Projects GPT-7.5 by Oct 2027 and Claude 7 by Apr 2028 — flowersslop · 2026-11-15
- Codex Pro users petition OpenAI to let unused weekly usage roll over — InTeGeR78 · 2026-11-15
- Rumored Gemini 4 Krypton spotted in Arena grey tests, nailing 20 art styles in one prompt — gaganghotra_ · 2026-11-15
- "Phonebook-trained models can't write Shakespeare": data distribution beats any harness — gerardsans · 2026-11-15
- Harness skeptic: model training-data support drives performance, not the harness — gerardsans · 2026-11-15