RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
Chang Guo, Yukun Xie, Bohan Tan, Zheng Chang, Zhaokai Yin, Qianli Ma, Yingqiao Wang, Chao Liang, Zhipeng Zhang
cs.RO
2026-09-22
RoboFollow makes language necessary via multi-task scenes. Nine VLAs score high at L0 then drop on L1-L3; Qwen3-VL, QA co-training, LangForce, and CFG all fail to close the gap.
High success rates on manipulation benchmarks often do not mean the policy read the instruction. In many RLBench, CALVIN, and LIBERO episodes the scene already admits one canonical action. Language then functions as a redundant task ID. A policy can score well while barely using it.
SJTU AutoLab calls this low scene entropy: given the initial visual specification, training task labels carry little uncertainty. Under the same task-label definition, the four LIBERO suites sit at 0.880 bits. RoboFollow hangs multiple kinematically feasible branches on each scene and raises that number to 3.782 bits. Vision alone cannot pick the intended branch.
RoboFollow is a diagnostic, not another leaderboard.
High scene entropy: 75 training task labels sit on six instruction-independent initial scene specs, with 16, 16, 16, 16, 7, and 4 labels per group. Training uses 3,750 demonstrations.
Four evaluation levels: L0 is in-distribution. L1 keeps the instruction and swaps object positions, catching policies that bound words to coordinates. L2 keeps the layout and recombines trained attributes, catching memorized text-object pairs. L3 perturbs both.
Scoring splits intent from motor skill. Intent Score asks whether the policy selected the instructed object, relation, waypoint, or logical branch. Execution Score asks whether the chosen subgoal was physically completed. A finishing stage is 20% of the score; extra actions after the task cost credit. Completion Rate is a looser binary.
Four scene families cover spatial prepositions, intrinsic attributes plus action primitives, trajectory and orientation constraints, and short logical forms (sequence, negation, observable if-then).
Nine policies: π0, π0.5, GR00T N1.6, OpenVLA-OFT, XVLA, ACoT-VLA, Lingbot-VLA, plus two world action models, Motus and FAST-WAM. Main runs fine-tune on all data. Mechanism studies fine-tune only 16 Scene 2 tasks (800 episodes).
In-distribution numbers can look saturated. π0.5 hits 99.1% and 100.0% Intent on Scene 1/2 at L0; π0 hits 98.2% and 95.6%; ACoT-VLA saturates Scene 1 L0 at 100.0%. Blanking the language instruction drops π0.5's overall mean Intent to 12.3%. The policy does use language in-distribution. That usage does not survive a layout swap.
| Model | Scene 1 L0 Intent | Scene 1 L1 Intent | Mean Intent |
| π0.5 | 99.1% | 45.5% | 55.7% |
| π0 | 98.2% | 0.0% | 48.4% |
| ACoT-VLA | 100.0% | 8.7% | 53.5% |
| GR00T N1.6 | 71.4% | 0.0% | 32.7% |
| OpenVLA-OFT | 9.8% | 0.0% | 10.7% |
On Scene 2, π0.5 falls from 100.0% at L0 to 34.2% at L3. Intent and Execution fall together, so the main failure is choosing the wrong thing to do.
A real-robot pilot with π0.5: 20/40 (50%) on eight training instructions, 6/40 (15%) on eight held-out instructions over the same objects. The two sets are not matched task pairs.
A 20-question visual QA probe on Scene 2: PaliGemma 1/20, pretrained π0.5 2/20, fine-tuned π0.5 3/20. Among questions the fine-tuned backbone answers correctly, about half the episodes still execute the wrong manipulation. Qwen3-VL-4B scores 19/20 on the same probe. Wired to a GR00T action head with QA co-training, L1-L3 still collapse. LangForce barely moves the needle. CFG at guidance 1.2 and 1.5 hurts even L0 (Intent 93.6% to 55.9% and 43.6%). In a high-entropy scene the unconditional prediction is already a multimodal guess, so the CFG residual is mostly noise.
Doubling demos per task from 25 to 50, raising instruction paraphrases from 1 to 5, and stretching training from 2k to 6k steps still leave L0-L2 / L0-L3 gaps near 56 and 48 points.
For anyone shipping a VLA: LIBERO-style success often measures that there was one reasonable thing to do, and the policy did it. Deployment scenes contain several graspable objects, paraphrased commands, and new layouts. RoboFollow builds that ambiguity into training. LIBERO-PRO perturbs at test time; the two diagnostics hit different layers.
Splitting Intent from Execution is the useful reporting habit. A single success bit cannot tell semantic error from control error. Code and data are released.
These numbers are after fine-tuning on this benchmark, not zero-shot generalist robots. The claim is that this adaptation recipe does not pass a high-entropy diagnostic. Broader data or a different training procedure are left open.
The authors are explicit. Objects are simple, horizons short, contact sparse, by design, to isolate language from motor confounds. The suite does not cover long-horizon planning, contact-rich manipulation, open vocabulary, or large-scale real-robot deployment. Controls never increase structural layout diversity at train time. The real-robot pilot uses unmatched instruction sets. No cross-simulator transfer is shown.
Two extra caveats. All nine models are fine-tuned on this data, so architecture and pretraining corpus are entangled; π0.5 beating OpenVLA-OFT is not an architecture result. L1's discrete swap is a clean kill for coordinate-memorizing policies. Real layout drift is usually smoother and noisier, so a 0.0% cliff is diagnostic, not a deployment forecast.