Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents
Ran Yan, Wei Fu, Jiale Li, Shusheng Xu, Zhiyu Mei, Jiaxuan Gao, Jiarui Zhang, Wentai Zhang, Hao Dai, Xujie Shen, Chuyi He, Zhen Pu, Jun Mei, Zhiyao Lin, Haitao Wang, Zhiqiang Ding, Jiawei Zhang, Huaijie Wang, Ruida Xu, Honghua Dong, Youhe Jiang, Yi Wu, Tongkai Yang, Binhang Yuan
cs.DC
2026-07-02
Position paper from the AReaL team: enterprise self-evolving agents are bottlenecked by RL systems, not algorithms. It proposes three pillars (a trajectory protocol, an enterprise data proxy, an evolution control plane) and grounds the weight-update branch in AReaL2.0.
LLM agents deployed in enterprises (coding assistants, customer-support bots, research assistants) are static. The model weights, system prompt, tool set, and in-context harness are frozen at deploy time. Making them better means a manual loop: collect data, fine-tune offline, rework the agentic loop, redeploy. An agent serving thousands of users generates a torrent of tool-mediated traces every day, and almost none of that experience feeds back into the agent.
Personal-scale self-evolving agents already exist; the paper repeatedly cites OpenClaw, which learns continuously from a single user's interactions. The gap this paper targets is moving that to enterprise scale (many teams, tenants, and compliance boundaries). The authors' position: at that scale, what's missing is not a bigger model, a cleverer prompt, or a better RL algorithm, but a system substrate that turns live interaction traces into governed, credit-assignable, replayable training material. This is a position paper from the AReaL team (Ant Group, Tsinghua, HKUST), and its thesis is that self-evolving agents are a systems problem as much as an algorithmic one.
Three co-designed pillars, each mapped to a specific gap.
First, an Agent Trajectory Data Protocol (ATDP). Existing agent logs capture prompts, completions, tool calls, latency, errors, and token usage: enough for debugging, not for training. ATDP defines a trajectory as a typed event sequence τ=(e1, e2, …, eT), each event et=⟨ot, ht, at, yt, rt, mt⟩: observable state (tool returns, retrieval snippets, user messages), hidden internal state (plan, scratchpad, confidence), the chosen action, its outcome, a reward signal, and metadata (latency, tokens, cost, tenant, model version, harness fingerprint). Drop ht and mt and you get a standard POMDP; keep them and you can carry LLM-specific artifacts such as reasoning traces, retrieval snippets, tool schemas, human corrections, and natural-language critiques. Two design points matter most: late-arriving rewards (a user correction in the next turn, a failing test, a delayed annotation) must be writable into the reward field after the fact without corrupting the original causal record; and every event carries versioned provenance (harness, tool version, retrieval-index snapshot, model checkpoint), because otherwise a trajectory is statistically useful but operationally non-reproducible.
Second, an enterprise data proxy. ATDP specifies what to record; the data proxy specifies how to capture it in production. Enterprises won't standardize on one framework (LangChain, CrewAI, OpenAI Agents SDK, Claude Agent SDK, MCP tools, custom orchestration), so the proxy intercepts at stable boundaries: model API calls, tool calls, retrieval, memory reads and writes, file or browser actions, human-approval events, final feedback. It does more than export traces. It converts production traffic into governed learning material, with redaction, access control, retention, and training-eligibility checks enforced before data enters the queue rather than bolted on after the fact. The distinguishing capability is replay: a monitoring trace can say "the agent called tool X and failed," but a training proxy must answer "would the agent have succeeded under a different prompt, model, memory, retrieval policy, or tool schema?"
Third, an agent evolution control plane. The central claim is that self-evolution is broader than weight updates. A deployed agent is a composite policy At=⟨πθ, Hψ, M, T, G⟩ (policy LLM, harness, memory, tools, governance). Different failures call for different surfaces: recurring missing facts suggest a memory insertion; tool-routing failures suggest a harness or schema edit; reusable procedural failures suggest a skill patch; a failure that persists across tenants, tasks, and tool configurations justifies a policy update. The control plane treats this as a governed decision problem, computing u=argmax JA(u | At, Dt) over an action set that includes weight updates, harness edits, memory updates, tool-schema changes, rollback, and no-op. Triggers are trajectory statistics (evaluator scores, user-correction rates, process-reward estimates, tool-failure clusters, canary deltas, cost per successful task, workload drift), not a human eyeballing two traces. Every intervention passes through replay-first evaluation, offline regression, tenant-aware safety checks, and versioned rollback.
This is a position paper. There are no benchmark numbers and no throughput or latency measurements. The three pillars are argued, not measured: whether the control plane's automatic triggering actually chooses "fix the memory" over "update the weights" correctly at real scale, and whether ATDP can be adopted across heterogeneous stacks, are questions the paper does not answer empirically.
The only concrete artifact is AReaL2.0, and it implements just one branch of the three pillars: online policy-weight updates. The approach reorganizes the existing offline RL framework AReaL into an agent-service-oriented online RL loop with four components. A Gateway is exposed as a drop-in replacement for the standard inference backend, so the agent points its LLM calls at it instead of SGLang or vLLM; a Router handles session affinity for multiple concurrent RL jobs; a Data Proxy records trajectories and stages training data; and an Agent-Compute Worker wraps rollout engines like SGLang and vLLM plus Megatron or FSDP training workers behind a micro-service interface, allocating compute to match the trajectory stream.
The motivating case is Nous Research's Hermes agent. Normally Hermes calls SGLang for model responses; swapping in an AReaL2.0-managed worker requires almost no change to the agent service, and the interaction stream is captured, recorded, and fed into the online RL training loop. The pitch is that you no longer build a separate RL environment to mimic production; the agent's native workflow is the training-data source.
For practitioners building agent platforms and post-training infrastructure, the direct value is that the paper makes the "live agent traffic to training data" pipeline concrete and composable, reusing existing inference and training stacks and localizing the change to swapping the inference backend. That is a different route from the current mainstream of training agents in synthetic environments offline and then shipping them static.
The bigger takeaway is the judgment itself. If enterprise self-evolution is bottlenecked on systems rather than algorithms, the place to invest is unglamorous infrastructure (trajectory protocols, data governance, replay capability, automatic intervention selection), not yet another RL algorithm. For architects and buyers, that is an actionable line for evaluation.
Be honest about scope. This is a gradual systems argument, not a capability leap. It offers no new model and no new benchmark record, and its value depends on whether the substrate actually gets adopted across teams and whether it can hold down compliance cost.
The authors scope themselves tightly. AReaL2.0 covers only the weight-update branch, leaving memory insertion, skill patching, harness editing, tool-schema evolution, replay governance, and automatic intervention selection as future work. ATDP stays at the "prototype formulation" stage with no real cross-framework deployment data, and the data proxy's multi-tenant isolation and federated training are design principles without measurement.
Several premises are unverified. The whole argument rests on the assumption that live trajectories are a better training source than synthetic or offline data, but the paper offers no downstream comparison to back that up; the premise is asserted, not demonstrated. The control plane's automatic triggering depends on a long chain of statistics (process-reward estimates, drift indicators, cost per successful task) whose noise and latency in enterprise settings go undiscussed, so "automatic decision" could easily degrade into moving human judgment into a threshold table. And putting redaction, compliance, and training-eligibility checks on the hot path before data enters the queue has unmeasured cost in latency and throughput, which in production is often the real blocker.