Freeform Preference Learning for Robotic Manipulation
Marcel Torne, Anubha Mahajan, Abhijnya Bhat, Chelsea Finn
cs.RO, cs.AI, cs.LG
2026-07-01
FPL trains a language-conditioned reward and reward-conditioned policy from annotator-named preference axes. Four real robot tasks average 75% success, 38 points above Filtered BC.
A robot that improves from its own rollouts needs a reward that is both dense and unambiguous. Binary success is too thin for long-horizon work. Setting a table means placing a large plate, a small plate, a cup, and cutlery; a single 1 at the end of a perfect episode teaches almost nothing about the middle. Hand-designed dense rewards are worse under image observations. Preference learning densifies the signal by asking which of two trajectories is better, but it collapses speed, safety, placement quality, and "did we smash the plate" into one overall label. On long tasks the two videos often win on different axes: one uses the spatula and drops the toast, the other grips the toast more cleanly and is unhygienic. Annotators then give noisy overall votes, and the learned reward is noisy with them.
Freeform Preference Learning (FPL), from Stanford IRIS, keeps those axes in language. People name the dimensions they care about and give pairwise preferences along each one.
Annotators watch two full-trajectory videos side by side and vote per axis. Ties are allowed and then dropped. Axes can be fixed in advance or typed in on the spot. Table setting uses placement quality for each item, formality, smoothness, carefulness, speed, and environmental damage. Plating toast is more open: 295 trajectory pairs, more than 1,477 axis-level labels, 41 distinct phrasings.
The real-world reward model is Qwen 3.5 VL 4B with a frozen vision encoder and full fine-tuning of the rest. Separate heads per axis would freeze the axis set and would fail on synonyms such as speed / fast / efficient. The prompt is "What is the score for {axis}?", followed by wrist and third-person frames at 128×128 with temporal striding. Each prefix of the trajectory gets a score; the trajectory reward is the sum, so the model can use history and localize long-horizon preferences in time. Training is still Bradley-Terry: on a given axis, the preferred trajectory must score higher.
Policy extraction does not average the axes into one scalar. A weighted sum is easier to hack and throws away composition. FPL picks a representative set of axes and conditions the policy on the axis names plus the trajectory's standardized rewards, written as text. The real-world policy is a fully fine-tuned π0.5 VLA (flow matching, action chunk 16, 8 actions executed per step, 30k steps, batch 32). At test time, changing the target rewards changes behavior with no retraining. Raw rewards are unbounded, so they are standardized per axis on the dataset and written into the prompt to one decimal place.
The loop can be one offline pass or iterative: roll out, collect more preferences, update reward and policy. Axes drift with competence. Early fold-shorts labels ask whether a fold happened; later ones ask about wrinkles and final alignment. Simulation is a different implementation: one head per axis, a float vector into a from-scratch flow policy, no language. Those sim numbers should not be read as language generalization.
Four real-world tasks on the DROID stack (Franka, two cameras, joint-velocity control), 20 rollouts per method.
| Method | Set table | Plate toast | Fold shorts | Cube in bowl | Mean |
| BC | 0.68 | 0.15 | 0.10 | 0.33 | 0.31 |
| Filtered BC | 0.65 | 0.30 | 0.25 | 0.29 | 0.37 |
| Single pref (match pairs) | 0.64 | 0.00 | 0.25 | 0.43 | 0.33 |
| Single pref (match labels) | 0.77 | 0.10 | 0.30 | 0.19 | 0.34 |
| FPL | 0.94 | 0.70 | 0.55 | 0.81 | 0.75 |
75% versus Filtered BC at 37% is the 38-point gap in the abstract. Matching the number of human labels, not just the number of video pairs, still leaves single-axis preference at 34% mean. Sparse-reward runs on table setting put objects in roughly the right places and often dropped plates instead of setting them down.
Simulation is a Robomimic fork, three seeds.
| Method | Rearrange success | Bimodal square throughput | Inverted throughput |
| BC | 0.04 | 0.71 | 0.25 |
| Single pref (match labels) | 0.79 | 0.73 | 0.74 |
| FPL | 0.84 | 1.19 | 1.24 |
On bimodal square the offline data has fast and slow left-peg demos and only slow right-peg demos. The target is a fast right-peg placement. FPL hits success 1.00 and time-to-success 253 steps, against 348 for demos and 392 for label-matched single preference. That "right peg, but fast" combination is not in the data. The same policy, conditioned for the left peg, reaches throughput 1.24; Filtered BC goes to 0.
Per-label annotation is about 3.4s for freeform axes versus 6.3s for a single overall preference, roughly 1.85× faster, because one video pair yields several labels.
This is a concrete change to robot preference learning, not a new algorithm family. The claim that holds is: when preference labels look noisy, the usual failure is collapsed axes, and pairwise comparisons can still work. For teams already iterating a VLA from offline data plus real rollouts, FPL is denser than success bits and cheaper than a hand-written reward. Changing the reward conditioning swaps bowls or pegs at test time without a new fine-tune. The fold-shorts curriculum, axes getting pickier as the policy improves, is a side effect of letting people type the rubric.
The Weighted Regression ablation is the other useful result. Average the multi-axis scores and extract the policy with advantage-weighted regression, and rearrangement success is 0.18. The reward model is not enough. Composition and test-time steering need the axes and the scores inside the policy.
The paper lists three: human preferences still cost more than unsupervised methods; test-time reward values are chosen by hand; the policy is conditioned on a fixed axis set, with variable-length axes left for stronger VLAs.
A few more discounts follow from the tables. Weighted Regression and Advantage Conditioning never appear in the real-world table, so the 38-point headline is versus Filtered BC, not versus every baseline on hardware. Twenty rollouts per task: fold shorts is 0.55±0.11, plate toast 0.70±0.10. Only plate toast lets annotators invent axes on the fly; table setting, fold shorts, and cube-in-bowl use predefined axes, so "freeform" covers less of the real robot suite than the abstract implies. Simulation uses K heads, not a language model, so composition and steering there are not evidence of linguistic transfer. Frames are 128×128 with a frozen vision encoder; how well the reward model scores wrinkles or hygiene is unknown, because the paper reports no standalone reward accuracy, only that checkpoints are picked on validation accuracy in the first 500 epochs. Dropping tie labels also throws away "both mediocre" pairs that would have helped calibration.