Partial RoPE study: rotating ~10% of dims matches full RoPE convergence with 10x smaller cache

BlancheMinerva · x · 2026-10-26

Presented at EMNLP 2026 in Budapest, the paper "Fractional Rotation, Full Potential?" (Max Planck Institute et al.) reports:

• Partial RoPE rotates only a fraction of query/key dims per head; existing models vary widely (GPT-OSS-20B: 16.9%, Phi-2: 40%, Llama family: 100%).
• The team pretrained 55 models from scratch (1B/8B, Llama/QK-Norm, FineWeb-Edu, 100B tokens, seq lengths 1024–8192, 0–100% rotation).
• Key finding: rotating 10% of dims reaches a loss band comparable to full RoPE while shrinking the RoPE cache up to 10x.
• NoPE (0%) destabilizes training with loss spikes; QK-Norm removes spikes but loss stays higher.
• Perplexity evals support this: LAMBDA perplexity 14.90±0.46 (NoPE) vs 11.86±0.39 (10%) vs 11.78±0.39 (100%).

Original post →

More from Research

• Speculative Diffusion Paper Maps the Speed-Quality Pareto Frontier via Two Tradeoffs — UCLA · 2026-10-26
• Git Records Fall Short for Auditing AI's Role in Research Repositories, Paper Finds — Koutian Wu · 2026-10-26
• What a 'Fused DeepLIFT' Actually Means: Gradient Overrides Without Hooks — jmschreiber91 · 2026-10-26
• New paper: AI agents negotiate badly under uncertainty and break deals on purpose — menhguin · 2026-10-26
• Cherimoya v0.3.1 Ships Fused DeepLIFT, ~7x Faster Attribution for Genomics Models — jmschreiber91 · 2026-10-26
• Proto-tools, a universal infrastructure layer for generative biology, now open source — jmschreiber91 · 2026-10-26

Research channel →