Forward-only LLM adaptation cuts memory and speeds up inference without backprop
机器之心 · wechat · 2026-11-20
Researchers from Lund University and Google DeepMind presented a NeurIPS 2026 paper on adapting LLMs using only forward passes. The method exploits differences in angular concentration of activations across SVD components to identify which weight matrices to low-rank, eliminating backpropagation entirely.
Compared to gradient-based LASER-Grad and exhaustive LASER search, the forward-only search achieves lower peak memory, higher speedups, and comparable accuracy across tasks like knowledge reasoning, hallucination detection, and fact verification. Experiments span GPT-J, LLaMA2, DeepSeek V1, Mistral, Qwen2 (including MoE variants) and multimodal Gemma4 up to 57B parameters; ablations show attention weight matrices can be skipped to further speed up the search.
The authors argue that with no gradient computation, low memory footprint and high speedup, the approach could enable real-time online adaptation during inference, letting systems adjust models to specific tasks or user groups at low cost. Paper: arxiv.org/abs/2611.10610.
More from Research
- Team claims Codex cracked 35 open Erdős problems in 2 months, some Lean-verified — IgorCarron · 2026-11-20
- ForgeSystem paper proposes AI agents forging demand-specific software — Div_pradeep · 2026-11-20
- Accord: open-source dense metric mapping from a single camera and IMU — BE2R · 2026-11-20
- AltRA-Test: ~100 samples suffice to decide if LLM judges can replace humans — Technion · 2026-11-20
- TwinD hits 95.6% AUROC on face synthesis deepfakes, beating prior detectors — Andrii Yermakov · 2026-11-20
- Watch a single artificial neuron compute: weights, bias, and one activation function — TinfoilTricorn · 2026-11-20