LoRA ranks move in lockstep early in training; higher rank means higher effective LR
thebasepoint · x · 2026-11-07
thebasepoint independently reproduced the same finding and discussed it with the original authors: at the start of training all LoRA ranks move in lockstep (for simple tasks), so increasing rank effectively raises the learning rate.
He cites a recent example: a spring paper found a subliminal-transfer effect with a U-shaped curve as a function of LoRA rank, which Lawrence Feng showed was caused by choices of alpha and dataset size.
Related event: No LoRA Learning Rate Can Match Full Fine-Tuning(4 posts)→
More from Research
- GLASS preprint teaser: graph-led LLM-assisted scoring of open-ended responses without human coding — GolinoHudson · 2026-11-07
- ICLR 2027 reviews mocked as worse than the papers, as LLMs improve mid-cycle — miniapeur · 2026-11-07
- LLMs can read and patch weights, but 20 fine-tuning steps erase the edits — kevinpl07 · 2026-11-07
- FlexiTac-Gen launches: AI-agent-assisted tool to design custom tactile sensors, accepted to CoRL 2026 — Haoyu_Xiong_ · 2026-11-07
- PolicyIK: NYU trains RL for inverse kinematics on a real arm, hitting 0.5mm median error — Lianhuiq · 2026-11-07
- Weight-space programming vs LoRA on 4 bugs: precise one-neuron patches that don't persist — kevinpl07 · 2026-11-07