Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements
Zhi Zheng, Rongsheng Chen, Yunpeng Ba, Zhenkun Wang, Yee Whye Teh, Wee Sun Lee
cs.LG
2026-08-18
Agentic ESOpt replaces backpropagation with full-parameter perturbations and reward-weighted updates, fine-tuning a 27B web agent at inference-level memory (8.41GB) and beating the strongest GRPO baseline by 12.5 points on 15-turn Sudoku.
RL fine-tuning of LLMs has proven itself on single-turn tasks and stumbles on long-horizon agents. Two hard spots:
Memory. Agentic RL (PPO, GRPO) backpropagates through whole trajectories, keeping activations, optimizer states, and a reference model resident. The paper's numbers: on Qwen3.5-4B, PPO needs 89.40GB and GRPO 58.88GB. Full-parameter RL on a 27B model is simply infeasible on four H100 80GB cards.
Credit assignment. With 15 turns and one terminal 0/1 reward, GRPO's group-relative advantage is trajectory-level and cannot tell which turn mattered; PPO adds a critic for turn-level advantages, but under sparse terminal rewards the critic never learns reliable values, and early advantage estimates are mostly noise. The theoretical point is plain: policy gradients sum H action-score terms, so estimator variance grows roughly linearly with H, while one ES perturbation maps to one whole trajectory and its variance carries no such H term.
Agentic ESOpt ports the 2017-era OpenAI evolution strategies into agent fine-tuning. Each step: sample G full-parameter Gaussian perturbations, roll each out for an environment reward, z-score the rewards within the population, and update parameters by the reward-weighted average of perturbation directions. Forward-only, no backprop.
The memory accounting: perturbations are never materialized as full parameter tensors; only random seeds are stored, and perturbations are reconstructed by in-place addition and subtraction (an established trick). Training memory therefore equals inference memory: 8.41GB for Qwen3.5-4B, 85.7% below GRPO.
Two design choices carry the paper:
Three experimental tracks:
Sudoku with controlled horizon (horizon strictly defined by the number of masked cells):
| Horizon H | Best RL | Agentic ESOpt |
| 5 | 90.63 (PPO) | 89.58 |
| 10 | 67.71 (GRPO) | 62.50 |
| 15 | 40.63 (GRPO) | 53.13 |
The ordering flips with horizon (PPO wins at 5 turns, GRPO at 10, ES at 15), which the authors rightly call more informative than a uniform win, consistent with the variance analysis. Under matched FLOPs the wall clock is also shorter (9.4h versus 19.0h for GRPO at H=15).
ReAct tool use and WebArena:
Test-time automatic heuristic design: inserting ES into EoH and plain sampling wins 28 of 36 matched comparisons.
The persuasive move here is not selling ES as a cheaper substitute but arguing it is the better-matched mechanism for long-horizon, sparse-feedback agents. Two direct engineering consequences:
The memory bill is rewritten. 8.41GB to train a 4B, four H100s for a 27B full-parameter. This puts agent fine-tuning within reach of small teams without waiting for open RL recipes from large labs. And prompt-space optimization (skill distillation, heuristic search) can now share rollouts with parameter optimization instead of running separately.
Boundaries apply: RL still wins on short horizons (PPO leads at H=5), and ES's edge grows only as the horizon does; when environment evaluation itself is expensive, the extra independent rollouts ES consumes can eat the saved backprop cost, a trade-off the authors flag themselves.
Self-reported:
From reading it: