Sebastian Raschka Adds Appendix on Positional Embeddings, Including YaRN and ALiBi

rasbt · x · 2026-11-19

Sebastian Raschka added an optional appendix to his article Understanding Positional Embeddings in Modern LLMs, restoring content originally trimmed from the draft and adding coverage of YaRN and ALiBi.

The appendix includes further explainers of token embedding lookups and attention calculations, additional discussion of experimental results comparing positional methods, and how long-context support is often improved by rescaling RoPE with YaRN. He recommends reading the appendix at points where it is referenced in the main text; a direct link is provided.

Original post →

More from Research

Research channel →