5 PEFT techniques explained: what LoRA, VeRA, Delta-LoRA actually do to model weights

techNmak · x · 2026-11-09

A well-structured thread (with a chart) explaining what five parameter-efficient fine-tuning techniques actually update inside the model:

• LoRA: freeze pretrained W, train two small matrices A and B (rank r ≪ d); update ΔW = BA, effective weight W' = W + BA — all adaptation happens in a low-rank space.
• LoRA-FA: freeze A too, train only B — half the trainable matrices, same core idea.
• VeRA: both A and B are frozen random matrices; you only train two tiny scaling vectors b and d that modulate their contribution — among the most parameter-efficient approaches.
• Delta-LoRA: fundamentally different — W is not fully frozen but updated via low-rank delta propagation: W^(t+1) = W^t + c(B A − Bt At).
• LoRA+: identical to LoRA except B gets a larger learning rate than A (ηB > ηA).

The unifying question: what is the minimum we actually need to learn to adapt a model? That is the essence of PEFT.

Original post →

More from Research

• Qdrant presents vector index-native relevance feedback in production at CIKM Rome — qdrant_engine · 2026-11-10
• Meta's LLM-generated failing tests catch 74 privacy SEVs, cutting incidents 20% — Meta_Engineers · 2026-11-10
• Toby Ord: time delays in the intelligence feedback loop rule out a true singularity — tobyordoxford · 2026-11-10
• Cornell professor recruiting Fall 2027 PhD students in probabilistic ML, AI4Science — dianarycai · 2026-11-10
• int8 quantization plus fp16 rescoring: 1.4M embeddings retrieved in under 2ms — tomaarsen · 2026-11-10
• int8 embeddings plus Usearch index: 1.4M vectors searched in under 2ms on CPU — tomaarsen · 2026-11-10

Research channel →