DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation
Haoyu Wang, Siyuan Qian, Yanjun Li, Zeyu Zhang, Yandong Guo, Boxin Shi, Hao Tang
cs.RO, cs.CV
2026-09-30
DexPolicy schedules exploration noise by training step instead of letting PPO learn it; real-robot dexterous Target success rises from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO).
RL for dexterous grasping has a built-in conflict: the action noise that helps a policy discover finger-object contacts is the same noise that ruins precise control once a grasp exists. The textbook answer is broad exploration early, narrow late. PPO treats the action standard deviation as a learned parameter, and in practice it barely moves. Across three mustard-bottle baseline runs, the logged scale ends at 0.185 to 0.186 after 5M steps, essentially the 0.20 it started from. Training finishes with near-initial exploration noise still attached.
This paper from Peking University and AI2 Robotics asks a narrow question inside ViViDex's trajectory-guided setting: if you take over that noise scale and make it an explicit function of training steps, does the final policy improve? It does, under three different policy-optimization settings.
The core move fits in one sentence: freeze PPO's learned standard deviation and replace it with a hand-designed scalar curve indexed by cumulative environment steps. Within a batch, the scale is fixed for sampling and for both likelihoods in the importance ratio. The curve was screened on mustard bottle and reused without retuning on the other objects, so the test is schedule transfer, not policy transfer.
Every paired comparison holds loss, architecture, reward, and optimizer fixed; only the noise rule changes. Both conditions are evaluated at the policy mean (zero evaluation noise), which separates learned control from noise sampled at deployment.
A component screen attributes the gain: noise scheduling alone lifts reward AUC from 19.96 to 27.36, while the tested update contraction alone drops it to 12.35. A shape gate at 95% AUC retention rejects uniform linear decay (lift retention 86.2%) and rejects removing the final drop to the plateau (lift 78.2%).
Simulation (ViViDex, five YCB objects, three seeds, deterministic Target success):
| Setting | Baseline | +DexPolicy |
| PPO | 32.0% | 35.7% |
| GRPO | 14.1% | 45.4% |
| FPO | 49.4% | 68.1% |
Hardware (RealMan RM75 arm with an Inspire/RH56 hand, three objects, 20 trials per condition, one model per condition, 360 trials total):
| Setting | Target baseline | Target +DexPolicy |
| PPO | 8.3% | 43.3% |
| GRPO | 10.0% | 63.3% |
| FPO | 25.0% | 85.0% |
Mean hardware Lift success also rises: 38.3% to 58.3% (PPO), 46.7% to 95.0% (GRPO), 45.0% to 93.3% (FPO).
Three counterexamples travel with the headline numbers. Under matched execution noise of 0.10, GRPO's advantage reverses (16.1% vs 8.7%), so deterministic gains do not establish perturbation tolerance. PPO's sugar-box return rises while final Target success stays at zero. And GRPO's mustard gain rides almost entirely on developmental seed 0 (baseline 0/0/92.5% vs 100/0.5/3.0% with DexPolicy); excluding seed 0, the five-object mean still climbs from 15.3% to 37.2%.
For anyone training manipulation policies with PPO-family updates, this is close to a free intervention: no loss change, no new module, just taking over a parameter the update rule was learning too slowly. The gain appears under three different updates and is largest on hardware (FPO up 60 points), which suggests slow learned-noise decay is a general property rather than a PPO quirk.
The second contribution is a measurement warning: training return, deterministic success, and noise tolerance are three outcomes that diverge, and a learning curve cannot distinguish them. Judge a schedule by terminal task success under the intended execution conditions, per task and per optimizer.
Honest framing: the effect size depends on the setting. PPO gains only 3.7 points in simulation, and there is no comparison against an equally tuned fixed low-noise baseline, so superiority over tuned alternatives is unproven.
The authors' own list: policies are object-specific, so no zero-shot generalization; three of eight PPO/FPO objects remain essentially unsolved; resetting, freezing, and annealing are not separated by ablation; hardware uses one model per condition on one platform, so 20 trials estimate that model rather than seed-level variance; the hardware simulator differs from the benchmark, so score gaps are not a controlled sim-to-real measure.
Two more from the tables: with three seeds, sample SDs run large (FPO on mug, 53.7 points) and the paper makes no significance claims; and the GRPO continuation bundles noise re-expansion with annealing in one intervention, so their separate contributions cannot be read off.