Why alpha shouldn't scale with rank: LoRA effective learning rate dynamics
thebasepoint · x · 2026-11-07
thebasepoint shares findings on LoRA learning dynamics: early in training all ranks move in lockstep, so higher rank means higher effective LR — hence the original alpha/r fix. Some practitioners instead scale alpha with r, making effective step size highly variable as a function of rank. The author exchanged insights with the original paper authors.
Related event: No LoRA Learning Rate Can Match Full Fine-Tuning(4 posts)→
More from Research
- GLASS preprint teaser: graph-led LLM-assisted scoring of open-ended responses without human coding — GolinoHudson · 2026-11-07
- ICLR 2027 reviews mocked as worse than the papers, as LLMs improve mid-cycle — miniapeur · 2026-11-07
- LLMs can read and patch weights, but 20 fine-tuning steps erase the edits — kevinpl07 · 2026-11-07
- FlexiTac-Gen launches: AI-agent-assisted tool to design custom tactile sensors, accepted to CoRL 2026 — Haoyu_Xiong_ · 2026-11-07
- PolicyIK: NYU trains RL for inverse kinematics on a real arm, hitting 0.5mm median error — Lianhuiq · 2026-11-07
- Weight-space programming vs LoRA on 4 bugs: precise one-neuron patches that don't persist — kevinpl07 · 2026-11-07