LoRA, QLoRA, VeRA and more: what each of 6 fine-tuning variants actually optimizes
techNmak · x · 2026-09-20
Everyone fine-tunes LLMs, but few understand what's actually being updated. LoRA, QLoRA, LoRA-FA, VeRA, Delta-LoRA and LoRA+ are not small variations of one method—each attacks a different bottleneck.
- LoRA: freezes W, represents the update as ΔW=BA; a d×d matrix goes from d² trainable params to 2dr.
- LoRA-FA: freezes A, trains only B—eliminating the need to retain input activations, cutting activation memory as rank grows.
- QLoRA: not "smaller adapters"—it quantizes the frozen base to 4-bit (NF4, double quantization, paged optimizers), fine-tuning a 65B model on a single 48GB GPU.
- VeRA: shares frozen random low-rank bases across layers, training only small scaling vectors.
- Delta-LoRA: uses the step-to-step change in BA to also update W, moving base weights without full fine-tuning's optimizer state.
- LoRA+: keeps the structure, but gives A and B different learning rates (B larger), since the two matrices optimize asymmetrically.
More from Research
- Podcast: How surgical data science teaches AI to understand what happens in surgery — ddonoho · 2026-09-20
- Dan Hendrycks Proposes 'Eigenism,' an Ethics Framework Making Human Flourishing AI Self-Interest — basedjensen · 2026-09-20
- Parallel structured LLM answers never check each other: the Zhaozhou MU problem — Successful-Farm5339 · 2026-09-20
- Advanced Matrix Factorization Jungle: A Living Map of Structured Factorization Algorithms and Phase Transitions — IgorCarron · 2026-09-20
- Sentence Transformers models quietly dominate Hugging Face's most-downloaded list — tomaarsen · 2026-09-20
- "Recipe for intelligence" paper published in Neuron — summerfieldlab · 2026-09-20