NoPE: Transformers without positional encoding beat RoPE/ALiBi on length generalization
BlackHC · x · 2026-11-03
Developer BlackHC resurfaced two positional-encoding papers: NoPE (Kazemnejad et al., arXiv:2305.19466) and BAM (Bianchetti et al.).
The NoPE paper systematically compares five PE schemes (APE, T5 Relative PE, ALiBi, RoPE, and no positional encoding) on reasoning and math tasks. Key findings:
- Commonly used methods like ALiBi, RoPE, and APE are not well suited for length generalization in downstream tasks.
- NoPE (no positional encoding) outperforms all explicit PE methods at zero extra compute.
- Theoretically NoPE can express both absolute and relative PEs; under SGD training its attention patterns mostly resemble T5's relative PE.
- Scratchpads are not always helpful for length generalization.
More from Research
- Stanford launches Digital Brain Project to build predictive models of the human brain, trained on HAI's Marlowe GPU cluster — StanfordHAI · 2026-11-03
- Qwen base model looped on 93% of MATH-500, casting doubt on claimed 6x gain — paul_cal · 2026-11-03
- After two silent grant rejections, researcher posts federated learning paper on arXiv — wandedob · 2026-11-03
- Scale AI's READY framework: identical-accuracy agents need 39.2% vs 29.6% human review — demian_ai · 2026-11-03
- LLMs can natively edit neural network weights: 0.016% of changes boost small model math 6x — Dr_Singularity · 2026-11-03
- Developer warns: do NOT use weight-space programming — djcows · 2026-11-03