AGZO paper: activation-guided zeroth-order optimization beats MeZO, fine-tunes Qwen3-4B where first-order OOMs
burny_tech · x · 2026-10-22
ICML 2026 paper AGZO (Activation-Guided Zeroth-Order Optimization) improves memory-constrained LLM fine-tuning:
- Problem: conventional ZO optimizers like MeZO estimate gradients via isotropic random perturbations, wasting evaluations on directions unrelated to meaningful updates
- Key insight: a linear layer's weight gradient lies in the subspace spanned by its input activations. AGZO extracts a low-rank activation subspace on the fly during the forward pass and restricts gradient estimation to those directions
- Theory: under spectral concentration assumptions, update directions provably have higher cosine similarity to the true gradient than isotropic perturbations
- Results: Qwen3-0.6B hits 89.2% on CB vs 80.3% for MeZO; enables Qwen3-4B fine-tuning on hardware where full first-order training OOMs, at near-identical peak memory to other ZO methods, significantly narrowing the gap with first-order fine-tuning
Paper: arXiv 2601.17261.
More from Research
- DeepMind AI Agent Autonomously Discovers and Lean-Verifies Rules for 11 Unsolved Turing Machines — rohanpaul_ai · 2026-10-22
- New Paper Derives Exact Stability Criterion for Outer Optimizers in Communication-Efficient Distributed Training — apsarathchandar · 2026-10-22
- Karpathy open-sources AutoResearch: AI agents that run the full research loop autonomously — mdancho84 · 2026-10-22
- MF² benchmark at EMNLP 2026 tests VLMs' movie comprehension with 850+ fact/fib pairs — andre_t_martins · 2026-10-22
- LLM literature priors boost Bayesian optimization: reaction yields jump from 6% to 39% — bravo_abad · 2026-10-22
- An 11-step AI workflow to extract data from papers for literature review — Faheem_uh · 2026-10-22