No LoRA learning rate can emulate full fine-tuning, finds agentic-experiments deep dive
thebasepoint · x · 2026-11-07
Due to quadratic effects in W = AB, there is no learning rate for LoRA that emulates full-parameter fine-tuning. The author notes that resolving contradictions in the literature is far easier now with agentic experiments — and highly valuable for both the experimenter and the community.
Related event: No LoRA Learning Rate Can Match Full Fine-Tuning(4 posts)→
More from Research
- GLASS preprint teaser: graph-led LLM-assisted scoring of open-ended responses without human coding — GolinoHudson · 2026-11-07
- ICLR 2027 reviews mocked as worse than the papers, as LLMs improve mid-cycle — miniapeur · 2026-11-07
- LLMs can read and patch weights, but 20 fine-tuning steps erase the edits — kevinpl07 · 2026-11-07
- FlexiTac-Gen launches: AI-agent-assisted tool to design custom tactile sensors, accepted to CoRL 2026 — Haoyu_Xiong_ · 2026-11-07
- PolicyIK: NYU trains RL for inverse kinematics on a real arm, hitting 0.5mm median error — Lianhuiq · 2026-11-07
- Weight-space programming vs LoRA on 4 bugs: precise one-neuron patches that don't persist — kevinpl07 · 2026-11-07