Observing and Controlling Features in Vision-Language-Action Models
Hugo Buurmeijer, Carmen Amo Alonso, Aiden Swann, Marco Pavone
cs.RO
2026-03-06
Linear observers and a closed-form controller steer four VLAs with no retrain. On DROID, π0.5 handle grasps rise from 14% to 74% and safe-object picks hit 50/50, at ~1% extra cost.
Vision-language-action models fold images, language, and proprioception into one policy that emits robot actions. They follow natural-language commands and transfer better than task-specific controllers. When the motion is wrong in the moment, there is almost no live correction. Prompt edits often leave the policy unchanged. Fine-tuning is expensive and can damage skills the robot already has.
Language models already have a cheap knob: add a small vector to a hidden state and the next tokens shift. Robots need that knob more, because the output hits real cups and real forks. VLAs are not drop-in copies of LLMs. They are multimodal, they run closed-loop, and many pair a Transformer with a flow-matching action expert. This Stanford and NVIDIA paper writes down feature-observability and feature-controllability, then tests four frontier VLAs: can a linear probe read state and action from pooled activations, and can a minimal shift along that probe change closed-loop behavior without a retrain.
Each Transformer layer is summarized by mean-pooling its tokens into a d-dimensional vector. A linear map W⊤x + b is trained to predict a pre-specified feature: end-effector pose, gripper opening, or the matching action command. Continuous targets use squared error; binary gripper labels use cross-entropy. Probes sit on the VLM prefix only. In hybrid models they never see the action expert.
Reading is not steering. The controller solves for the shortest additive perturbation u that puts the probe reading inside an interval [ξmin, ξmax]. With a linear observer and an interval target, u has a closed form: push along W if the reading is past the bound, do nothing if it already sits inside. That same u is then broadcast to every token at the layer. The paper shows this broadcast is the unique minimum-Frobenius-norm way to realize a given pooled shift.
At inference the extra work is one matrix-vector product and one add per chosen layer. Weights stay frozen.
The sweep covers autoregressive OpenVLA and π0-FAST, plus the hybrid π0 and π0.5, on LIBERO, DROID, and BridgeData V2. Cartesian state probes reach R² 0.81 to 1.00 and orientation 0.53 to 0.99. Action commands decode much worse. π0.5 is strongest on continuous features on average. Action R² drops by about a third from LIBERO to DROID; state scores barely move. The sim-to-real gap on actions is larger than the gap between the two architecture families.
Closed-loop tests freeze π0.5 in LIBERO and on DROID hardware (one RTX 5090). Baselines are the unmodified policy, prompting, value-vector steering, rejection sampling, and action clipping.
| Setting | Base | Prompt | Value vector | This method |
| Wine-bottle base grasp (sim, 100) | 28% | 40% | 3% | 73% |
| Coffee-mug handle (hardware, 50) | 14% | 46% | 38% | 74% |
| Banana not metal fork (hardware, 50) | 52% | 40% | 64% | 100% |
| Glass not plastic bowl (hardware, 50) | 48% | 62% | 94% | 100% |
On transport-height bounds, rejection sampling at K=100 still stays under 10% constraint satisfaction and inflates median inference time by 583%. Value vectors hit about 74% on the upper bound and 0% on the lower bound. The closed-form intervention adds about 1% time over 50 forward passes. Action clipping, as expected, works well for these geometric limits; the lower bound still needed a 4 cm margin so the arm starts correcting early enough.
This is a runtime dial on a frozen VLA: change how it grasps, or which object it reaches for, without new demos and without fine-tuning. 1% overhead fits on the same 5090 that runs the policy. Prompts barely moved these behaviors. Value vectors made the wine-bottle preference worse, 28% down to 3%. Adapting the step to each sample, along a direction that actually decodes the feature, is more reliable than adding a fixed vector.
It is preference alignment and soft constraints, not safety. The paper is explicit: 50/50 safe-object picks still do not certify safety, and an output-side filter remains complementary. For pure geometry, clipping the action may be the more honest tool.
Observers need labeled rollouts; this work fits them on 50 base-policy trajectories. Large robot datasets often lack per-dimension labels of this kind. Interventions touch Transformer layers only, so the flow-matching action expert is neither interpreted nor controlled. There are no formal bounds from a representation-space bump to the executed trajectory, which bars using this layer alone in safety-critical stacks.
Preferred grasps are not perfect: 73% on the bottle, 74% on the mug handle. The remaining rollouts still grab the wrong place. Figure 6 shows lift constraints rising with task success still high, but the text does not quote exact satisfaction percentages for the proposed method. Hardware tasks use 50 rollouts each, so the 100% object-selection figures should be read as 50 of 50 in this setup, not as a guarantee.