Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning
Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang, Yongliang Shen
cs.AI, cs.CL
2026-08-24
Agent-G2 samples hint depth from an online Gaussian. Qwen2.5-1.5B/7B reach 95.3%/98.4% ALFWorld success at under one-third the probe rollout cost.
Long-horizon agents often get a single terminal reward after dozens of steps, so on-policy exploration from the start state rarely succeeds. Hint-based RL keeps a prefix of an expert trajectory and lets the policy roll out from a state closer to success. How much prefix to keep is the whole game: too little and successes stay rare; too much and rewards saturate so advantages collapse.
Prior methods treat that depth as a deterministic scalar. A shared schedule ignores that a two-step Pick and a twenty-step Pick Two sit in the same batch. Per-sample binary search or enumeration fits one task, at the price of extra probe rollouts. Shared schedulers put more than 60% of assignments outside the informative band; driving mismatch near zero with enumeration needs about 20× the GRPO rollout budget.
A diagnostic shows useful depths occupy a band around the per-task optimum. Aligning Bernoulli variance by distance to that optimum yields a Gaussian fit with σ=0.22 and R²=0.92. The job is to cover the band, not to pin a point.
Agent-G2 clusters tasks offline by expert-trajectory length. A global baseline μglobal steps up or down with batch success versus a 0.5 target. Each cluster keeps EMA estimates of mean success Ak and variance Vk. A task's center is the global baseline plus λ(0.5−Ak); its spread tracks within-cluster variance. One ratio is drawn from that Gaussian and converted to a prefix shared by R rollouts of the task. The policy is updated with GRPO plus teacher-forced loss on the sampled prefix. The same rollouts refresh the Gaussian, with no probe budget and no learned depth predictor.
On ALFWorld, Qwen2.5-1.5B reaches 95.3% success and the 7B model 98.4%. That is +1.5 / +2.3 over the strongest hint baseline, +3.9 at both scales over the strongest hint-free RL, and +7.4 / +6.6 over RLVMR, the strongest Aux-RL method. Versus probe-based Enumeration the gaps are +9.3 / +2.3, without probe rollouts. On WebShop both scales score 92.3 reward, with 78.9% / 84.4% purchase success. 1.5B Agent-G2 at 95.3% already beats 7B BEACON at 94.5%.
| Method (1.5B / ALFWorld) | Success |
| Full SFT | 56.3% |
| GRPO | 72.8% |
| BEACON | 91.4% |
| Target acc (best schedule) | 93.8% |
| Enumeration | 86.0% |
| Agent-G2 | 95.3% |
Dropping stochastic sampling to the mean cuts success to 89.8%; a variance-matched uniform draw gets 88.3%; collapsing clusters to K=1 gets 89.1%; removing GRPO leaves prefix SFT at 26.6%. Wall-clock is about 88s per step versus 57–80s for schedules and 285–425s for probing (3.24–4.83×).
Hint depth on long-horizon agents should not be a global knob or an expensive search. Rollouts already paid for policy updates are enough to cover the learnable band by difficulty cluster. A 1.5B model with this schedule beats larger hint-free baselines, so prefix allocation can substitute for backbone scale in sparse-reward household and shopping settings.
Training needs one expert trajectory per task; without demonstrations the method does not apply. The Gaussian is motivated by the informativeness profile on these two benchmarks; multimodal or skewed depth profiles may want another family, though a uniform-draw ablation suggests covering the band matters more than the exact shape. Clusters are frozen by trajectory length and do not track changing relative difficulty as the policy improves. Evaluation stays on ALFWorld and WebShop, with no direct comparison to the math-reasoning hint literature.