Q-Planning: robot policies self-improve from 25% to 80% success, no new demos

rbhar90 · x · 2026-08-27

The demo-to-deployment gap in robotics is a data problem: the thread argues additional teleop SFT data on robot foundation models suffers from covariate shift, and ICL alone doesn't reach mastery. Q-Planning offers an alternative via test-time self-improvement.

A frozen large visuomotor BC policy is paired with a small off-policy Q-function
At inference, the policy samples N candidate action chunks; the Q-function scores them and a single-step Q-weighted average is executed
Both successful and failed rollouts go into a replay buffer that fine-tunes only the Q-function; the BC policy is never updated
On a hard fine-grained manipulation task, success rises from 25% to 80% in 100 robot attempts (30 minutes), with no extra human data

Essentially a "thinking mode" for robot policies via test-time compute.

Original post →

Q-Planning: robot policies self-improve from 25% to 80% success, no new demos

More from Embodied