Tsinghua NeurIPS Paper: First-Order Action Supervision Boosts Offline RL, Especially With Little Data
量子位 · wechat · 2026-10-10
A NeurIPS 2026 paper from Tsinghua AIR, 'Higher-Order Action Supervision Makes A Strong Policy Class,' argues that jittery robot control and overreactive driving stem from abrupt changes between consecutive actions, yet standard policy learning only supervises single-point actions. Without changing the policy network, the authors add a first-order constraint: the policy's directional derivative along state trajectories should fit the action difference a′−a in the data.
The idea uniformly covers deterministic (TD3+BC), Gaussian stochastic (via Ornstein-Uhlenbeck temporal correlation), and flow-matching policies, and injects first-order information into value learning through a continuous-time HJB framework — adding gradient-alignment terms on top of the Bellman residual — retrofitting TD3+BC, ReBRAC, IQL and IFQL.
Experiments span 50 OGBench tasks and 10k-transition D4RL sets: ReBRAC jumps from 26 to 61 on antmaze-giant-navigate; on walker2d-expert, TD3+BC goes from 2.7 to 82.3 and pure behavior cloning from 26.7 to 91.1. Rollouts show smoother action changes. The takeaway: local variation in continuous trajectories is especially valuable when samples are scarce, as a plug-and-play loss term.
More from Embodied
- MIT CSAIL's X-Hinges 3D-Prints Everyday Objects With Built-In Sensing Across Three Motion Axes — MIT News AI · 2026-10-23
- Formic raises $50M Series B to scale physical AI across 100+ US factories — garrytan · 2026-10-23
- fal starts fine-tuning a physical robot to draw, results next week — OdinLovis · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- CU Boulder's ATLAS Institute Opens Tenure-Track Robotics Professorship — ryosuzk · 2026-10-23
- MiTaS multi-resolution tactile imitation learning hits 80% success vs 35.5% vision-only baseline — Jan_R_Peters · 2026-10-23