Meituan's AutoRL runs RL training unattended with a multi-agent crew, +20 points on Qwen3-4B AIME24

AutoRL: A Multi-Agent System towards Autonomous Reinforcement Learning

Simon Tang, Wenjie Shi, Yongjing Yin, Keheng Wang, Jialin Liu, Zheng Zhang, Linvo Lin, Xiangyuan Xue, Kan Ren, Jingang Wang, Xunliang Cai

cs.AI, cs.LG

2026-11-13

AutoRL supervises RL runs in-flight with multi-agent roles, fixing graders, tuning entropy and recovering crashes. Qwen3-4B beats static GRPO on all 9 benchmarks, +20 on AIME24.

What problem this solves

RL training does not survive being left alone. As the policy updates, the distribution of generated experience shifts and the learning signal drifts with it. Distributed execution adds numerical and operational failures, and a grader with subtly wrong rules can look healthy for dozens of steps before performance bleeds away. Existing autonomous-research agents organize work across complete experiments: run one, read the results, design the next. Human RL researchers intervene while the run is live, repairing a grader the moment reward curves look wrong, or rethinking the recipe when a plateau drags on. AutoRL, from Meituan with CUHK, ShanghaiTech, and Fudan, hands that in-flight supervision to a multi-agent system, and frames automated RL as a cornerstone of recursive self-improvement (RSI).

Method

The system splits an RL researcher's job into three role types, each with its own context, coordinated through shared records:

Two design choices carry the load. Monitoring needs frequent checks and fast responses while research needs long, quiet analysis, so sharing one context would make them compete; and because testing an RL hypothesis burns GPU hours, cheap validation such as code inspection and trajectory replay happens inside deliberation, leaving GPU budget only for approved interventions.

Results

Baselines are a frozen GRPO recipe (the standard algorithm for LLM RL) and a native Codex general agent driven by the same gpt-6.1-sol model.

SettingComparisonMetricResult
Qwen3-4B math (DAPO-Math-17K)Static GRPOAIME 202446.67% → 66.67% (+20.00)
Qwen3-4BStatic GRPO, general agentNine benchmarksBeats both on all nine
Qwen3-4B plateau studyOwn plateau bestAccuracy29.74% → 41.58%
LongCat-Flash-Lite, flawed recipeUnmodified GRPOMath eval73.16% vs 56.05%
LongCat coding agent, 14.7 daysStarting modelSWE pass@152.29% → peak 65.13%

The flawed-recipe run is the cleanest demonstration. LongCat-Flash-Lite trained on math with a simplified grader and no KL regularization: the static recipe slid from 65.53% to 56.05%, while AutoRL applied six hot updates over 64 steps, four grader repairs (option mapping, multi-value matching, LaTeX parsing, unit normalization) plus KL coefficients raised from 0.001 to 0.01, and finished at 73.16%, a 17.11-point gap with zero restarts.

In the plateau study, AutoRL left the config untouched for the first 100 steps while collecting evidence, and even deferred an entropy proposal that lacked support. It then repaired answer-option binding via a cold resumption from checkpoint 100, raised the entropy coefficient from 0.001 to 0.004 in stages, and added a -0.02 penalty for incorrect responses meeting a strict looping criterion. Accuracy first cleared the plateau at step 135 and reached 41.58% at step 180, 11.84 points above the plateau best.

The coding-agent run lasted 352.5 hours (about 14.7 days) on 28×8 GPUs and absorbed 11 runtime failures (seven NaN crashes, one dtype mismatch, three worker or communication faults), each recovered from a checkpoint. AutoRL also tightened log-ratio clipping from [-13, 5] to [-5, 3], cut the learning rate from 5e-6 to 2e-6, and enabled FP32 accumulation in the MoE components; when FP32 residuals triggered a new dtype failure, it reverted only that option and kept the rest. SWE pass@1 finished at 63.20% with a 65.13% peak.

At a matched budget of 273 steps against the general agent: 41.58% versus 32.11%, 131 initial candidates versus 21, and a 20% reversion rate versus 50%. The general agent spent 44.7% of its steps retraining after checkpoint rollbacks; AutoRL had zero rollbacks and withdrew its four changes through online control, keeping model and optimizer state intact. Swapping the supervising backbone to glm-5.2 dropped the peak to 32.89%, still above the general agent. At the 1.7B scale, AutoRL beats the static recipe on eight of nine benchmarks with one tie, while the general agent edges it on AIME 2024 by 1.67 points.

Why it matters

The expensive part of RL training is babysitting: NaN crashes at 3 a.m., quiet reward hacking, plateaus nobody notices for days. This paper shows that current LLM agents, given the right role split and RL-specific tools, can absorb most of that watching and local repair; 14.7 days unattended with 11 self-recovered failures is the hardest evidence. The HOT/COLD execution paths, evidence review, and Borda voting form a blueprint teams building automated training supervision can borrow directly. The boundary is equally clear: every gain came from tuning, grader repair, and local reward refinement. This is an engineering-integration win, not a new algorithm.

Limitations

The authors are blunt about scope. Of 131 proposed candidates, 110 (84%) targeted grading and data refinement, and all three attempts at learning-rule redesign died in deliberation. They observed no conceptual leaps and call this the major unresolved challenge on the path to RSI. Swapping to GLM-5.2 cost 8.69 points of peak accuracy, so the ceiling tracks the supervising model. No multi-seed variance is reported, and some benchmarks have small denominators, such as MBPP at 6.79% for Qwen3-4B and LiveCodeBench at 2.00% at the 1.7B scale, where a few points can be noise. The +20 on AIME is measured against a static recipe, not a human-tuned best one. The token cost of running gpt-6.1-sol or glm-5.2 as supervisors is not disclosed.

Terms

Source

What people are saying

Related papers

All paper explainers