Due to quadratic effects in W=AB, no LoRA learning rate emulates full fine-tuning
thebasepoint · x · 2026-11-07
thebasepoint goes deeper on LoRA learning rates: due to quadratic effects in W = AB, there is no learning rate that makes LoRA emulate full-parameter fine-tuning.
He explains the original recommendation was to fix alpha in alpha/r, but in some contexts people started scaling alpha with r, making the effective step size highly variable as a function of rank.
Related event: No LoRA Learning Rate Can Match Full Fine-Tuning(4 posts)→
More from Research
- MIT and Stanford release free robotics textbook merging classical engineering with AI — GioeleZardini · 2026-11-07
- UIUC releases OpenFrontiers: open research problems corpus plus scientific rediscovery benchmark for LLM agents — Jiarui Yao · 2026-11-07
- Dinuc shuffling preserves the CA initiator dinuc, a pitfall for seq2func genomics models — anshulkundaje · 2026-11-07
- Wrong negative set confounded TF effects: ProCapNet team explains missed GATA motifs — anshulkundaje · 2026-11-07
- AI model predicts and explains cell-type-specific TSS usage driven by MEF2 motif — anshulkundaje · 2026-11-07
- Stanford trains ProCapNet on 200+ ENCODE PRO-cap datasets to map human transcription initiation — anshulkundaje · 2026-11-07