LoRA, LoRA-FA, VeRA, Delta-LoRA, LoRA+: 5 parameter-efficient fine-tuning methods explained
mdancho84 · x · 2026-11-04
Modern fine-tuning often adapts an LLM without touching billions of pretrained weights. This guide breaks down 5 parameter-efficient techniques with diagrams and formulas:
- LoRA: freeze W, learn a low-rank correction ΔW = BA
- LoRA-FA: also freeze A, train only B — fewer trainable params
- VeRA: freeze random A and B, learn tiny scaling vectors only
- Delta-LoRA: let W evolve too, guided by the low-rank delta update
- LoRA+: same LoRA but with different learning rates for A and B (ηB > ηA)
A solid mental map for data scientists moving toward AI engineering.
More from Research
- DR Pixel: open-source 27B deep research agent reads paper pages without OCR, matches GPT-5.4 — CShorten30 · 2026-11-04
- Nigerian Legal Corpus released: cleaned constitutional and legal texts for RAG and legal NLP — saheedniyi_02 · 2026-11-04
- Vulnerability research is a topology problem: reusing local proofs across audits — dyn___ · 2026-11-04
- Agents will soon edit their own weights and memory — but who studies when to use which? — sarahcat21 · 2026-11-04
- Hand-writing all weights of a 100k-param model beats training it at spotting fake legal citations — AaronBergman18 · 2026-11-04
- Stanford reran 50 famous LLM findings on 20 models — 44% no longer hold — heyshrutimishra · 2026-11-04