LeRoPE makes RoPE frequencies learnable, beating RoPE across 52M-2.5B scales
eliebakouch · x · 2026-10-30
A UCSD lab paper (Karypis, McAuley et al.) proposes LeRoPE, treating RoPE's fixed rotation frequencies as learnable scalar parameters instead of hyperparameters. Training model ladders from 52M to 2.5B parameters from scratch, they observe an emergent high-norm positional frequency band; LeRoPE consistently beats RoPE and partial RoPE at all scales, with RoPE needing 3.4% more FLOPs to match it at the largest scale. eliebakouch notes the name is easy to misattribute to a major lab.
More from Research
- Know, then Draw: UCSD & Adobe Cut Image-Gen Training Compute 48x While Boosting DPG-Bench 10% — Yucheng Mao · 2026-10-30
- Live mechinterp session hunts 'neurons of sadness' in Qwen3.5 — burny_tech · 2026-10-30
- Stanford prof Anshul Kundaje: most papers don't need formal review, the system is overwhelmed — anshulkundaje · 2026-10-30
- RT-SAFE Benchmark Shows Faster Reasoning Doesn't Mean Safer Agents in Real-Time Embodied Environments — Lianhuiq · 2026-10-30
- LeCun team's LeHami: symplectic Hamiltonian flow keeps JEPA world models from collapsing latent states — ylecun · 2026-10-30
- RLNF: Training AI on Human Preferences Purely From Brain Signals — alex_verem · 2026-10-30