LoRA's first step isn't standard SGD: an implicit low-rank preconditioner throttles early fine-tuning
pbaylies · x · 2026-09-07
A technical thread argues that LoRA does not perform standard SGD on your weights.
- At step 0, LoRA runs gradient descent distorted by an implicit low-rank preconditioner: P = alpha^2 A^T A.
- This random bottleneck quietly throttles early-stage fine-tuning gradients.
- The takeaway: understanding LoRA requires accounting for this preconditioning structure rather than treating it as plain low-rank SGD on weights.
More from Research
- Computer-Use Models Are Still 'Low-Frequency, Highly Batched' — Minecraft May Stay Unsolved Until 2030 — mike64_t · 2026-09-07
- IBM's STAIR uses tables of contents for generative retrieval, hitting 82.6% Recall@1 — omarsar0 · 2026-09-07
- MinHash classic: clustering huge datasets with a KV store in five lines — moultano · 2026-09-07
- Safe RL With Stability Guarantees: Learning Without Ever Falling Down — tomssilver · 2026-09-07
- Principia benchmark: video models score 0.8 on VBench but under 0.42 on physics consistency — CSProfKGD · 2026-09-07
- 1-bit quantized embeddings cut vector index storage up to 60x with <1% quality loss — burkov · 2026-09-07