Debating Whether Positional Encodings Are Truly Needed for Causal Models at Scale
mgostIH · x · 2026-08-09
The discussion centers on the mechanics of linear attention (like DeltaNet) and positional encodings (like RoPE). The quoted post suggests RoPE can be viewed as a product of accumulating transition matrices, which in linear attention can generalize into data-dependent matrices that change at each position.
The replier counters this, arguing that the entire theoretical wall of text might be a delusion: perhaps positional encodings are simply unnecessary for causal models at scale, regardless of the specific architecture used.
Related event: Mathematical Analysis of Kimi K3: Why RoPE Is Dropped(2 posts)→
More from Research
- Simplex Diffusion Models paper talk by EPFL/DeepMind's Justin Deschenaux on Oct 19 — ArnaudDoucet1 · 2026-10-16
- Meituan LongCat team's Data World synthesizes data environments to train data agents — meituan-longcat · 2026-10-16
- FDE-Gym benchmark shows huge gap on real AI delivery tasks: top model scores 0.563 — DeepLethe · 2026-10-16
- ForcingBooster halves computation of autoregressive video diffusion with patch-wise denoising — VAST-AI · 2026-10-16
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- SemEval-2027 Task 9 opens: multimodal news framing analysis across 4 languages — preslav_nakov · 2026-10-16