PoLoRA Optimizer Accelerates LoRA Finetuning, Boosts Stability
deepcohen · x · 2026-08-19
PoLoRA is a Preconditioned Orthogonalized LoRA optimizer addressing Adam's ignorance of LoRA's matrix structure. By combining product-aware spectral updates, curvature preconditioning, and magnitude rules, PoLoRA achieves the final loss of tuned Adam in 1.2-1.7x fewer steps on code and math tasks. It is also less sensitive to learning rates, with stable optimal rates across ranks.
More from Research
- FAR framework enables AI to solve 1977 Erdős–Straus conjecture — wellecks · 2026-08-19
- Study reveals LLMs are surprisingly bad at controlling robot arms — tom_doerr · 2026-08-19
- AlphaFold performance exceeds expectations on TNFα — MoAlQuraishi · 2026-08-19
- Neuroscience Research: Handwriting Activates Widespread Brain Connectivity — CurieuxExplorer · 2026-08-19
- Expert rebuts Anthropic: Binder design is not a drug proxy — rapha_gl · 2026-08-19
- Fun: Qwen Model Learns to 'Piggyback' on Walls in Snake Game — mervenoyann · 2026-08-19