RoboFollow Benchmark Exposes Instruction-Following Mirage in 9 VLA Policies
AutoLab-SJTU · hf · 2026-09-23
AutoLab at SJTU released RoboFollow, a diagnostic benchmark showing embodied agents' instruction-following ability is far weaker than success rates suggest.
- Root cause: low scene entropy — scenes admitting only one valid task make language redundant, so policies score high while barely using instructions.
- Design: high scene entropy (multiple kinematically distinct task branches per scene), a four-level (L0–L3) perturbation protocol probing spatial relations, attributes, trajectory constraints and logic, and confound-controlled Intent/Execution scoring to separate comprehension from motor execution.
- Findings: across 9 VLA and WAM policies, strong L0 performance fails to transfer to L1–L3; mitigations including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance all fail to close the gap.
- Code and dataset are open-sourced on GitHub and HuggingFace.
More from Embodied
- Homebrew robot arm build log Day 7: geartrain CAD, wiring and gripper sizing — _Stocko_ · 2026-11-11
- Perceptron AI launches EgoHome, an open benchmark for sub-second egocentric hand understanding — rohanpaul_ai · 2026-11-11
- Runway open-sources Praxis-1, turning video pretraining into real robot control — runwayml · 2026-11-11
- Unitree G1 humanoid wearing VR goggles pilots an A1 drone in land-air robot demo — CyberRobooo · 2026-11-11
- CoRL Spotlight Meta-Ctrl predicts robot plan failures, Llama 3 8B beats GPT-5.5 on VirtualHome — guyvdb · 2026-11-11
- Luna, a jewelry-style AI wearable that speaks before you ask, launches on Kickstarter — nikola_mr64990 · 2026-11-11