Agent Priors-guided Policy Learning
Puming Jiang, Tianrun Hu, Haozhe Du, Yibo Li, Zhiwei Xue, Xinhu Li, Harold Soh
cs.RO
2026-09-29
APPL uses each skill's structural prior to shape policy training and as runtime docs for composition. 89.6% OOD success from 2 demos; hiding the docs cuts composition from 8/16 to 2/16.
A robot trained on a handful of demonstrations gets deployed against new goals, reordered operations, and objects in new places. That takes two kinds of generalization at once: compositional generalization, recombining learned skills into new tasks, and skill generalization, keeping each skill's policy working in states it never saw. The two are coupled. Task-level scheduling is only as good as its knowledge of what each skill can physically do, and a skill only matters if it survives the states that scheduling drops it into.
The interface between the levels is where information leaks. Where a skill works is decided by the structure its policy was trained with, but composition sees only a name, an instruction, or a symbolic operator. VLA models fold semantics and control into one network and leave generalization conditions implicit; TAMP connects the levels through symbolic preconditions and effects, hand-written or learned; LLM agents that call policies as tools know what a skill is called without the physical conditions under which it works, and handoffs are a documented failure mode. The closest precedent, SymSkill, co-invents predicates, operators, and skills from demonstrations, but locks its structure into relative-frame predicates and dynamical-system controllers.
APPL (Agent Priors-guided Policy Learning, National University of Singapore) uses each skill's structural prior in two roles. A structural prior states what a behavior depends on, for instance that a grasp depends only on the gripper's pose relative to the object. Implemented in training, through the representation, action parameterization, or an auxiliary loss, it shapes where the policy generalizes. Written out in language, together with training support, handoff conditions, and verification evidence, it becomes the interface a runtime agent reads to judge where the policy applies.
The choice that matters: policy structure stops being a fixed engineering decision and becomes a per-skill hypothesis. The library keeps several implementations of the same skill with different inductive biases, and the runtime agent chooses among them.
Experiment 1, six MetaWorld tasks, skill generalization only; OOD averages recombined (C) and extrapolated (E) states:
| Method | N=2 OOD | N=20 OOD | Mean over N |
| Diffusion Policy (B0) | 28.96% | 46.67% | 37.86% |
| Fixed relational prior (B1) | 37.92% | 56.25% | 47.29% |
| APPL, first proposal (q1) | 56.04% | 78.75% | 67.92% |
| APPL (q4, with feedback round) | 89.58% | 93.13% | 92.40% |
With two demonstrations, 89.58% against 37.92% for the fixed prior. At N=20 every method hits 100% IID while extrapolated-state success is 25.42% for B0 and 88.33% for q4, so the gain is generalization, not fitting. The first proposal alone beats B1 by 18.1-22.5 points, selecting among three candidates adds 10.2-24.0, and the feedback round adds 2.9-12.1 more. Feedback can backfire: door at N=20 drops from 100.00% to 96.25%.
Experiment 2, five long-horizon ManiSkill tasks with twelve demonstrations each (full-task policies take no goal input and enter the shifted-object comparison only):
| Method | Shifted objects (M) | Task-level variants (T) | Unseen compositions |
| Full-task Diffusion Policy | 10.0% | – | – |
| Full-task + single LLM prior | 10.0% | – | – |
| Agent + VLA (fine-tuned pi0.5) | 32.5% | 70.0% | 6/16 |
| APPL | 50.0% | 92.5% | 8/16 |
| APPL, all interface info hidden | 20.0% | 65.0% | 2/16 |
| APPL, fixed-rule scheduler | 22.5% | 52.5% | 1/16 |
The ablation row carries the argument. Same frozen library, interfaces hidden: shifted-object success falls from 50.0% to 20.0% and unseen compositions from 8/16 to 2/16. Restoring only verification scores and exit criteria recovers 45.0%, so verification evidence carries most of the shifted-object benefit, while the prior, handoff, and support descriptions add most on task-level variants (72.5% to 92.5%). One worked example: a return policy's handoff note describes an emptied container, but its prior excludes particle state, so the agent inferred a full container would also work and finished in two calls.
For anyone building skill libraries, the transferable claim is small and concrete: do not discard the structural assumptions a policy was trained with. They double as interface information, and on the same frozen policies, visibility into them is worth 30 points of success on shifted objects. Against SymSkill, the prior family is no longer fixed in advance; against a fine-tuned pi0.5 driven by the same runtime agent, this route holds up better in the few-demonstration, shifted-object regime (50.0% vs 32.5%).
Boundaries worth stating: both experiments run in simulation on structured state vectors rather than images (Exp. 1 policies consume no images; the designing agent may view up to 16 per demonstration set). This is an incremental step on a known route. The contribution is the interface argument; the backbone is a standard Diffusion Policy.
The authors' own list: a structural prior states intended generalization and guarantees nothing, so a policy can still fail outside its demonstrated support; the demonstrations contain no recovery trajectories, so a dropped object stays dropped; verification runs only from demonstrated entry states and can misrank policies under shift, which on covered peg assembly pushed down a policy that was actually useful and dropped that task from 8/8 to 1/8, offsetting gains on the other four tasks.
More caveats from the read. q4 over q3 mixes an added candidate with development-set selection, so the feedback effect is never isolated; the authors say so in the appendix. Demonstrations come from scripted experts with no failures or recoveries. The six Exp. 1 tasks had been seen before this study during earlier development. Runtime depends on GPT-5.6 Sol at extra-high effort with up to 128 requests per episode, and the reproducibility statement is blunt that re-running the LLM agents yields different designs. Cost and latency of the runtime loop go unreported.