LeRoPE Beats Standard RoPE Across Scales With Just 32 Extra Parameters
burny_tech · x · 2026-07-30
An ICML paper introduces LeRoPE (Learnable RoPE Frequencies), noting that LLMs typically rely on hand-picked RoPE frequencies for token positioning. Letting the model learn these frequencies yields better results.
- Minimal overhead: With just 32 extra parameters, LeRoPE outperforms standard RoPE across different model scales.
- Dominant positional band: The learning process naturally discovers a dominant positional band around 2.2x the training context length.
More from Research
- AI Labs May Crack ARC-AGI-3 in Weeks After Opus 5 Breakthrough — Angaisb_ · 2026-07-30
- Single 8x 5090 Rig Hits 167k tokens/s Training Throughput, Beating DDP — jon_durbin · 2026-07-30
- Why Hasn't Anyone Tested Replacing Most Attention Layers with ResNet MLPs? — kalomaze · 2026-07-30
- Predicting AI Loss of Control: Tsinghua & Cambridge Propose Behavioural Framework — 机器之心 · 2026-07-30
- Human-in-the-Loop Hyperspectral Imaging Optimizes UAV Landmine Detection — RITiger · 2026-07-30
- GPT-5.6 Produces Lean-Verified Proof for New Ramsey Number Bound — naval · 2026-07-30