ACLArena: Agent Continue Learning in Multi-stage Post-training
Haixin Wang, Xiaoxuan Wang, Junkai Zhang, Han Zhang, Renliang Sun, Alexander K Taylor, Yidan Shi, Haoran Deng, Chenguang Wang, Jason Cong, Yizhou Sun, Wei Wang
cs.AI
2026-09-21
Sequential Math→Search→E-commerce→IF on Qwen3-8B drops AIME from 23.3 to 6.0. MLE freezes an SDFT backbone plus per-domain LoRA-RL experts, nearly matching specialist oracles.
Industrial agent post-training is staged. Math reasoning first, then a search tool, then a multi-tool retail desk, then instruction following. Technical reports for Qwen3, GLM-5, and DeepSeek-V4 all do some version of this. Almost none of them measure, under one controlled stack, how much of each skill is still there at the end.
UCLA and UC Santa Cruz turn that into a four-stage lab. The base model is Qwen3-8B-Base. The curriculum is fixed: Math → Search → E-commerce → IF. Math and IF go straight to critic-free RL (GRPO-style). Search and e-commerce need a rejection-sampled SFT cold start so the model learns tool syntax, then RL. Search uses one tool; e-commerce uses up to fifteen. Math trains on DAPO-Math-17K, search on Natural Questions, e-commerce on τ³-Bench retail.
Sequential training does not stack. Seq-Math lifts AIME26 (avg@16) from 6.46 to 25.83 and NQ from 13.3 to 22.0 as a side effect. Seq-Search takes NQ to 45.2 while AIME holds at 23.33. The e-commerce stage then wipes both: AIME 6.04, NQ 14.6, multi-hop search 9.4 down from 37.4. The final IF stage scores 84.8 on IF-Eval, but only recovers NQ to 33.5 and AIME to 10.21. No peak survives into the last checkpoint.
The mechanism is directional. Each stage pulls weights toward its own single-task oracle; the four oracles point only partly the same way, and later checkpoints sit farther from those oracles. Token-level scoring of a frozen Seq-Math rollout shows the rewrite is not uniform: about 94% of low-entropy positions stay put, and almost all of the drift sits in the remaining 6% high-entropy decision tokens, the places where the model picks a query or a tool.
Three industry recipes are reimplemented under the same eval, then a fourth is assembled from the diagnosis.
Their own recipe is MLE (Mixture of Low-Rank Experts). SDFT first packs multi-domain behavior into a shared backbone, then that backbone is frozen. Each stage gets its own LoRA, trained with RL in its environment, so residuals cannot overwrite each other. At inference the agent execution context selects the expert: system prompt and tool schemas are already visible at serving time. No benchmark name or gold task label is required.
The design follows three measurements. Task updates are only partly aligned, so one weight set trades one skill for another. SFT moves parameters about 3 to 7 times more than RL, which is good for installing behavior and dangerous for old skills. RL updates are local enough to live in a low-rank residual. The same oracle-mixture SFT from the base model and from the sequential endpoint has cosine 0.77; unrelated SFT tasks sit at 0.05.
All numbers are Qwen3-8B-Base on two H200 nodes, mean ± std over three runs. In-domain: AIME26, NQ, τ³-Retail, IF-Eval. Out-of-domain: GPQA-Diamond, MMLU-Redux, single/multi-hop search, τ³-Telecom/Mock, IF-Bench.
| Method | AIME26 | NQ | τ³-Retail | IF-Eval |
| Math expert | 25.83 | 22.0 | 2.3 | 43.3 |
| Search expert | 9.79 | 49.9 | 2.7 | 62.4 |
| Sequential final | 10.21 | 33.5 | 29.6 | 84.8 |
| +MMOPD | 21.25 | 45.2 | 27.7 | 84.6 |
| +SDFT | 22.29 | 48.3 | 22.3 | 53.6 |
| +uniform merge | 14.79 | 39.0 | 11.8 | 74.6 |
| MLE | 21.04 | 49.7 | 32.9 | 85.0 |
No shared-weight method wins everywhere. MMOPD is the least bad compromise: AIME 10.21 → 21.25, NQ 33.5 → 45.2, IF-Eval almost unchanged, retail slightly down. SDFT restores multi-hop search from 25.0 to 38.0 and wrecks IF-Eval (84.8 → 53.6) and τ³-Telecom (45.9 → 21.7). Uniform merge tops MMLU at 80.3 and loses the most on e-commerce and IF.
MLE sits next to the specialists in-domain: NQ 49.7 vs Search expert 49.9, τ³-Retail 32.9 vs 33.2, IF-Eval 85.0 vs 86.2. AIME 21.04 still trails the Math expert's 25.83, but it is far above sequential. Out of domain it posts the table's best GPQA (42.4), single-hop (57.7), multi-hop (38.6), and τ³-Mock (66.7).
A single-teacher ablation supports the token story. After e-commerce has erased search, Mixed OPD from the search checkpoint lifts student NQ from 14.6 to 48.4, above the teacher's 45.2, while τ³-Retail falls from 34.0 to 15.8. Search is not only tool syntax; it also needs the reasoning that later stages overwrite.
For teams stitching staged agent post-training, this is a controlled comparison, not a new loss. Qwen3, MiMo-V2, GLM-5, Nemotron-Cascade 2, and DeepSeek-V4 all close with some form of OPD or distillation. Here OPD, offline replay, and weight merging sit on the same Math→Search→e-commerce→IF course. The hard result: inside one shared weight set, recovering one skill costs another.
If serving already knows the environment (retail desk, search session, exam), MLE's router is almost free and each stage adds one LoRA. If the product must be a single weight file with no environment switch, MMOPD is the least damaging shared-model option in this paper. SDFT is a cold start, not a finish.
SFT updates are about 3 to 7 times larger than RL. Behavior cloning rewrites the model; RL polishes nearby. That matches a lot of lab intuition; this paper plots the displacement for agent stages.
The authors flag two. Everything is Qwen3-8B-Base on one fixed curriculum; other families, other orders, and longer pipelines are untested. MLE picks an expert from observable environment context. Ambiguous context, mid-episode environment changes, and episodes that need two specializations at once are not measured.
A few more gaps sit in the design. Routing moves task identity out of the weights and into the scheduler, which is a weaker claim than "one deployable agent." There is no learned router, and LoRA rank, extra parameters, and latency are not reported. The curriculum is frozen by prerequisite, with no permutation study, so "e-commerce is the destructive stage" may be "tools jump from 1 to 15 after search," not a property of retail itself. AIME26 is avg@16 with three-run std, but MLE still trails MMOPD by 0.21 on AIME; the win is on the two agent tasks, not math versus the Math expert.