Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models
Zhaokai Yin, Zhipeng Zhang
cs.RO
2026-08-04
VLA robots collapse on paraphrased instructions. The cause is architecture, not comprehension: re-binding task semantics with a frozen T5 lifts LIBERO-Para success by up to 44.6 points.
Vision-language-action (VLA) models map images and language instructions to robot actions and look strong on standard benchmarks. They have an embarrassing failure mode: rephrase "pick up the red object" as "grasp that red thing" and the success rate craters. The authors test three top VLAs (VLA-Adapter, SmolVLA, pi0.5); paraphrasing the instruction costs up to 67.53 points. The usual fix is to throw data at it, training on many synonymous rewrites. That works but is expensive. The starting point here is a counterintuitive finding: the models do understand.
The authors run a diagnostic first. On LIBERO-Goal (ten tasks sharing one visual scene, so only language distinguishes them), they generate actions from a paraphrase and check which canonical instruction's correct action it lands closest to (Retrieval@1). All three score above chance, so task semantics survive inside the features. A causal intervention sharpens the point: during a paraphrase run of VLA-Adapter, they swap only the language feature entering the final action block for its canonical-run counterpart, leaving every other input untouched. Success jumps from 60% to 96%, and 96.8% of the action gap disappears. The semantics are intact; the loss happens in translating them into actions.
Why do equivalent instructions produce different features at all? Two controls answer this. Replacing the live image fed to the auxiliary language branch with a fixed dummy raises paraphrase success from 46.82% to 61.58%, because encoding the dynamic visual input together with the text amplifies feature drift. Separately, the authors estimate 32 directions that separate canonical from paraphrased outputs and remove them, lifting closed-loop success from 55% to 90%; rewording adds a separable, systematic shift that the action policy happens to be brittle to.
The fix is Grounded Semantic Re-binding (GSR). A frozen T5-large encodes the instruction alone, with no image and no robot state, so the semantics stay stable. A projection then injects this representation into the model's native multimodal fusion point, and the action expert is trained from scratch. Training uses only the ten canonical-instruction tasks, no paraphrases. The governing principle: language must be encoded independently of the current frame first, then reintroduced to interact with vision and state inside the fusion pipeline.
The injection point depends on the architecture. VLA-Adapter replaces its native input with a fixed sentence ('perform the task') to force reliance on T5. pi0.5 already generalizes across paraphrases, so it keeps its native input and uses T5 as a supplement. SmolVLA injects the T5 features into SmolVLM's language positions so they interact with the scene before action generation.
On LIBERO-Para (4,092 paraphrased episodes):
| Model | Native Full Para | +GSR | Gain |
| VLA-Adapter | 46.82% | 70.94% | +24.1 |
| SmolVLA | 4.47% | 49.12% | +44.65 |
| pi0.5 | 73.60% | 75.59% | +2.0 |
pi0.5 + GSR reaches a PRIDE score of 70.4, edging past the previous best Xiaomi-Robotics-0 (69.2). A control rules out the idea that the model simply has more parameters: adding the same trainable capacity to VLA-Adapter without T5 leaves paraphrase success stuck at 46.82%.
A conflict test exposes a hazard. When a model receives both its native input and T5, which one drives the robot? Feeding the wrong instruction to VLA-Adapter's native channel crashes success from 47.31% to 5.11%; feeding it to T5 only drops it to 44%. The native channel is unreliable yet hijacks control, which is exactly why GSR neutralizes it.
On a real robot, native VLA-Adapter collapses to task-independent motion in the shared scene and scores 0% under both canonical and paraphrased instructions; GSR scores 50% and 40%.
This moves the problem of a VLA that cannot follow language from the data column to the architecture column. For practitioners: if your VLA is unstable under rewording, audit where task semantics get tangled with vision and where they get translated into actions before scaling the corpus. GSR is a light intervention. The frozen T5 runs once per task and, once cached, adds no per-step inference cost, yet it approaches models trained on far more data. It also hands over a transferable diagnostic recipe (layer-wise feature replacement to locate the bottleneck) that is not specific to these three models.
The authors concede two points. First, ParaVLA, the fully decoupled alternative (T5 for language, DINOv2 for vision, fused only at the end), trades near-perfect paraphrase robustness for weak scaling: upgrading the visual backbone does not deliver VLM-level gains. Second, GSR reinitializes the action expert, so on pi0.5's large action module the canonical success dips from 93% to 91% under the standard training budget and needs twice the steps to recover to 96%; an efficient initialization scheme for large experts is still missing.
Another concern stands out: the real-robot study covers only six tasks on one model, a small sample, and LIBERO-Para's rewrites are limited to action phrasing and object reference. Freer natural language (multi-step, omission, ambiguous anaphora) is out of scope here.