LeRoPE Beats Standard RoPE Across Scales With Just 32 Extra Parameters
burny_tech · x · 2026-07-30
An ICML paper introduces LeRoPE (Learnable RoPE Frequencies), noting that LLMs typically rely on hand-picked RoPE frequencies for token positioning. Letting the model learn these frequencies yields better results.
- Minimal overhead: With just 32 extra parameters, LeRoPE outperforms standard RoPE across different model scales.
- Dominant positional band: The learning process naturally discovers a dominant positional band around 2.2x the training context length.
More from Research
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16
- Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
- Not Boring #215: Cammarata's Clean Networks Claim, Tesla's Flying Roadster, Tau Cut 73% — Not Boring (Packy McCormick) · 2026-10-16