Sebastian Raschka Explains Positional Embeddings in Modern LLMs, from GPT-2 to RoPE
Ahead of AI (Sebastian Raschka) · rss · 2026-11-15
Sebastian Raschka published a deep technical walkthrough of how LLMs encode token position information, with visual examples, math, and PyTorch code. It covers why explicit positional info is needed beyond causal masks, learned absolute embeddings in GPT-2, a long-awaited RoPE tutorial, and a gallery of positional encoding choices across 2026 open-weight models (e.g., MiniMax-M3 applying RoPE in every block). It serves as a foundation piece for his upcoming LLM architecture trends analysis.
More from Models
- "Phonebook-trained models can't write Shakespeare": data distribution beats any harness — gerardsans · 2026-11-15
- Harness skeptic: model training-data support drives performance, not the harness — gerardsans · 2026-11-15
- Security veteran: traditional tooling is obsolete, run an agentic hacker against your own systems — Kyrannio · 2026-11-15
- AIFlow: China looks set to save local AI — AIFlow_ML · 2026-11-15
- All major AI models appear to have stopped using the em dash — mark_k · 2026-11-15
- Same 3D gummy orange prompt, two models: Fable 5.2 vs Opus 5.5 side by side — FinanceYF5 · 2026-11-15