SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation
Saksham Singh, Zheyuan Hu, Max Sobol Mark, Jeffrey Yu, Zackory Erickson, Aviral Kumar
cs.RO
2026-09-19
SeeQ trains a PaliGemma Q-function on the active subtask, decodes it in language, then best-of-N steers π0.5. Four real bimanual tasks rise from 35.4% to 66.7% success.
Generalist robot policies still break on long-horizon work. Multi-stage jobs stack small errors; precision jobs need retries on the same stage. Ranking action chunks with a Q-function and picking the best of N is the usual patch: the base policy proposes, the critic selects.
Sparse full-task rewards make that critic hard to train. Park et al. found Q-estimation error can grow nearly exponentially with horizon in offline RL. Monte Carlo regression skips bootstrap noise, then pays with high-variance targets glued to the behavior policy. TD can score actions better than the data, but long Bellman backups on thinly covered teleop traces are unstable.
Manipulation already has a shorter target sitting inside the rollout. Even when the task reward is minutes away, the robot is usually chasing a nearby objective: get the hanger through a sleeve, retry a failed latch. Scoring the active subtask needs a shorter horizon than scoring full-task success. SeeQ trains the critic on that shorter target.
The backbone is 3B PaliGemma. Pretraining uses the bimanual slice of RoboCOIN, 131 tasks and about 40k episodes. Each downstream task then finetunes both the critic and the base policy π0.5. Train and test sample N=8 action chunks from the base policy and execute the highest-Q chunk.
The prediction target and the interface both change.
Nearby frames can belong to different subtasks, a failed latch versus the recovery pose. Implicit inference yields jumpy values. Explicit decode pins the current objective in text. The backup action is the best of N base-policy samples under the target Q (TD-BoN), matching test-time steering and fitting values under an improved action distribution.
Four real bimanual tasks at 60 Hz, on dual xArm-7 and dual YAM, 24 trials each:
| Task | Data | Base π0.5 | SeeQ steering |
| shirt-hang (386 eps) | RaC corrections | 10/24 | 22/24 |
| lid-sealing (447 eps) | RaC | 10/24 | 15/24 |
| grocery-packing (473 eps) | expert teleop | 9/24 | 17/24 |
| LEGO-disassembly (97 eps) | expert teleop | 5/24 | 10/24 |
Average success moves from 35.4% to 66.7% (1.88×). Shirt-hang and lid-sealing use RaC data, where the policy runs until it starts failing and a human takes over. Grocery-packing and LEGO-disassembly are pure expert demos. LEGO has only 97 episodes from a single starting configuration; SeeQ still moves 5/24 to 10/24. On qualitative traces, Q drops after a failed latch or a missed Pringles grasp and recovers after the retry. Task-level TD is nearly flat across the whole trajectory.
Same backbone, same pretraining, same finetune data, different targets:
| Objective | shirt-hang | grocery-packing |
| Base policy | 10/24 | 9/24 |
| Task-level Monte Carlo | 9/24 | 10/24 |
| Task-level TD | 12/24 | 9/24 |
| Subtask-level SARSA | 13/24 | 10/24 |
| SeeQ | 22/24 | 17/24 |
Task-level MC and TD barely move the base policy. Subtask-level SARSA keeps subtask rewards and language decode but bootstraps with the dataset’s next action, and the gain is small. Both the subtask target and the best-of-N backup are doing work.
Robot pretraining is not optional. On shirt-hang, skipping it and finetuning PaliGemma directly yields 8/24, worse than the base policy. A task-specific ResNet-50+MLP critic with the same TD-BoN recipe reaches 16/24, still below full SeeQ at 22/24. On 6 held-out trajectories (12,582 frames), median image-gradient norm is 0.093 for SeeQ, 0.323 without robot pretraining, 0.494 for ResNet-50. The no-pretrain critic is even less sensitive to actions (median 0.058 vs 0.093), so it cannot separate candidates.
Subtask supervision is equally sharp. On shirt-hang, dropping prediction and conditioning yields 8/24. Adding the prediction loss without feeding the text to the value head yields 15/24. Full decode-and-condition is 22/24.
This moves the generalist-policy recipe onto a critic: a VLM backbone, large robot data, and language as the scaffold for intermediate state. If a stack already runs a VLA such as π0.5 and has subtask-labeled offline data, test-time ranking can raise long-horizon success without retraining the policy.
The gain is incremental. The Q-function only ranks candidates the base policy already proposed. Every target task still finetunes the critic. Training still wants subtask labels; a VLM can generate them when they are missing, but the main experiments use existing annotations.
Training needs subtask labels, and slicing recoveries or boundary frames into discrete spans is ambiguous; the right granularity is task-dependent. A wrong subtask decode at test time scores actions against the wrong objective. Maximizing the current subtask can ignore downstream damage, such as knocking over the next object while packing the current one. Evaluation only selects among a frozen policy’s chunks. There is no experiment that uses this Q to improve the policy, and no low-data or zero-finetune transfer result.
The trial budget is small: 24 rollouts per task, four tasks, two platforms. Lid-sealing only moves 10/24 to 15/24. Task-level MC on shirt-hang is 9/24, slightly below the unsteered policy, so a bad critic can hurt. Dropping robot pretraining to 8/24 shows an internet-pretrained VLM is not a drop-in value function.