What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
Xingshan Zeng, Zishan Xu, Boju Zhang, Yuzhou Wu, Lingzhi Wang, Jianghao Lin, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Weinan Zhang, Yong Yu, Qun Liu, Weiwen Liu
cs.AI, cs.CL
2026-08-27
Huawei and SJTU treat agentic data as (E, q, τ, v) under ACE: keep only valid samples, then place mass by learner-relative difficulty and non-redundant coverage.
LLM agents learn to call tools, edit files, and act in simulators from interaction traces. Those traces are not ordinary instruction-response pairs. An environment, a task, a rollout, and a success signal have to stay consistent. A fluent dialogue can still be useless if the task is infeasible, the tool implementation disagrees with its schema, an observation does not follow from the previous action, or the verifier rewards the wrong terminal state.
The literature is organized by domain: tool use, repository repair, GUI, simulators, scientific discovery. That split mixes two questions: how candidates are constructed, and how they are verified or selected. Related mechanisms appear under different names, while datasets built by different processes are scored with metrics that do not line up. This Huawei Noah and Shanghai Jiao Tong survey offers a shared ledger for comparing those pipelines.
Agentic data is written as a four-tuple $(E,q,\tau,v)$. $E$ is the actionable world (tools, state, transitions, observation interface). $q$ is the task signal. $\tau$ is a realized interaction. $v$ is an optional verifier. Serialization can differ; the four roles cannot.
Generation paradigms are grouped by anchor and dependency order. Forward generation builds an environment, then tasks, then trajectories. Grounding is strongest here, and a narrow environment constrains everything downstream. Reverse generation includes task-first (specify a capability, then invent the world and the trace), trajectory-first (explore, then write the user-facing task), and structure-first (tool graphs, blueprints, or plans before surface realization). Adaptive systems feed verification failures and learner feedback back into earlier factors.
ACE treats generation as constrained distribution design. Accuracy is an admission gate: local validity plus consistency among the four factors. Inside that support, Complexity places mass relative to a declared learner and execution setup (model, scaffold, tools, inference budget). Harder is not automatically better. Diversity is a batch property that rewards coverage and penalizes redundancy. The unit is a relation that changes what an agent can observe, decide, and do, not a renamed tool or a paraphrased request.
This is a survey, with no new head-to-head experiment. The literature shows three migrations.
Accuracy moves from plausibility judgments to execution: real function calls, database states, compilation and tests, proof assistants. APIGen checks format, then execution, then semantics. EnvFactory-style work argues that a smaller set of robustly verified environments can compete with simply growing the environment count.
Complexity moves from static heuristics (length, tool count, opaque wording) to learner-relative difficulty. Useful samples often sit in a band the base model cannot solve reliably, but an agent-assisted configuration can. Graphs, withheld information, and environment policies can add burden only when the missing pieces are recoverable and executable.
Diversity moves from sample count, domain labels, and surface rewrites toward behavioral coverage. A larger API table or more paraphrases does not expand support if it still induces the same observe-decide-act relation.
The discussion pushes ACE into earlier training stages and self-evolving loops. Pre-training and mid-training need not store a complete four-tuple per example; local transitions, inverse dynamics, and reachability targets can still teach interaction priors. In a self-evolving loop, generation is no longer offline stockpiling. Failures and coverage holes decide what the agent should experience next.
If agent-data work is still competing on trajectory count, this paper changes the ledger to three questions: can this sample execute, is it near the current model's frontier, and is it a new behavior or a duplicate. SFT still teaches format and RL still optimizes policy, but once generators, verifiers, and learners co-evolve, a frozen offline dump is not enough.
The engineering order is concrete. Harden environments and verifiers before scaling tasks. Tune difficulty with the model, instead of a fixed "longer is harder" template. Measure diversity so that strategy coverage is not confused with rollout noise.
The survey does not offer a unified ACE benchmark; most claims are a taxonomy of prior work. Execution-based accuracy can still be gamed, and safety, intent, and long-horizon side effects will not show up in a unit test. The "learnable band" for complexity depends on protocol and thresholds, with no portable default. Diversity still lacks a reusable cross-domain metric; entropy and paraphrase distance inflate easily. In a closed self-evolution loop, generator, learner, and verifier can drift together. The paper calls for independent anchors; current systems almost never make that hard.