DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attack
Hoseong Tae, Jong-Seok Lee
cs.CV, cs.LG
2026-08-04
DRIFT stickers the robot gripper and attacks only the first denoising step of π0 and π0.5, breaking nearly all LIBERO tasks and far beating action- and embedding-space baselines.
Vision-language-action (VLA) models like π0 and π0.5 do not generate actions token by token. They use flow matching: starting from Gaussian noise, they integrate a learned velocity field over several steps until a continuous action chunk falls out. A 2026 robustness benchmark reported that these flow-matching VLAs shrug off adversarial perturbations far better than autoregressive VLAs such as OpenVLA, and the takeaway was that flow matching is inherently harder to attack.
Hoseong Tae and Jong-Seok Lee at Yonsei University argue this robustness is an artifact. Existing attacks either perturb the final action space (UADA) or the vision encoder's embedding space (EDPA), and both ignore the multi-step denoising ODE that actually produces the action. Miss that trajectory and you miss the weak point.
DRIFT's threat model is deliberately conservative. The attacker pastes one 32×32-pixel patch (about 2–3 cm, roughly 2% of the wrist image) on the robot's gripper, optimizes it offline once, and leaves it fixed. At deployment there is no model access and no online feedback; the rest of the observation, the proprioceptive state, and the language instruction stay untouched.
What DRIFT attacks is the denoising velocity field, not the final action or the embedding. π0 integrates noise into action over K=10 Euler steps, and the authors test each step k for exploitability. The vulnerability is front-loaded: corrupting any early step (k≤5) breaks essentially every task, while corrupting a late step (k≥6) collapses to near-zero success. The reason is that an ODE solver is autoregressive. A directional error introduced at step k rides through every later Euler update, so the earlier the injection, the longer the amplification chain.
The counterintuitive part follows. If every early step works, you would expect attacking several of them to be at least as strong. It is the opposite. Widening the window from step 1 to the first 3 steps drops average success from 99.8% to 77.5%; the first 5 steps give 79.7%. The cause is a gradient conflict specific to input-space optimization. The patch gradients from different denoising steps do not point the same way: the first-step gradient g₀ and the last g₉ have cosine similarity about −0.35, nearly opposing. Summing fighting gradients cancels the signal, and the more steps you add, the worse the cancellation (normalized alignment falls from 1.00 at M=1 to 0.45 at M=10).
So attacking only the first step is both the strongest and the cheapest option. It skips the remaining nine steps in the forward and backward pass, cutting optimization cost to 1/K of a full-trajectory attack.
Across four LIBERO suites, π0 succeeds on about 97–98% of tasks when clean. A random patch barely registers (0.7%). The strongest prior attack, EDPA, fails outright at the same 32-pixel budget and needs a 64-pixel patch to reach 25.3%. UADA averages 13.2%.
DRIFT, at 32 pixels, hits 99.8% average attack success and the highest action discrepancy:
| Method (attacking π0) | Pixels | Avg success | Avg NAD |
| UADA (action space) | 32 | 13.2% | 4.3% |
| EDPA (embedding space) | 64 | 25.3% | 5.4% |
| DRIFT | 32 | 99.8% | 8.5% |
π0.5 is tougher and needs a 64-pixel patch, but DRIFT still reaches 99.3% average success and 20.0% NAD, against UADA's 30.5% and EDPA's 1.5%.
The failure mode is consistent. The authors call it "phantom grasp": under the patch the gripper closes within the first 7 control steps (mean 3.8), long before reaching anything, clenching on empty air. The arm keeps moving (mean displacement 40.6 cm, never closer than 18.6 cm to the object) but never completes the reach-then-grasp sequence the task requires.
The perturbation budget ε shows a sharp phase transition: success stays below 8.2% for ε≤0.05, then jumps to 97% at ε=0.1. All or nothing.
For anyone deploying π0 or π0.5 in the real world, the implication is direct. An adversary with gradient access to the deployed policy and one inconspicuous sticker can make the robot grasp at nothing almost every time. The 2026 benchmark's "robust" verdict was an artifact of attacks aimed at the wrong surface, not a property of flow matching.
It also points the defense. The earliest denoising steps are the soft spot, so protecting the opening of the trajectory matters more than guarding the whole chain. The patch can even be constrained to grayscale and blended into a dark gripper without losing effect, at about twice the perturbation budget, which makes physical concealment plausible.
The threat model is white-box, and cross-model transfer is weak. A patch optimized on π0 transfers to π0.5 at only 8.9% (and 24.7% the other way), so an effective attack needs gradient access to the specific deployed policy and cannot be ported across model versions. That limits the black-box risk.
All experiments run in LIBERO simulation, with no real-robot validation under realistic lighting or camera noise. The phantom-grasp failure is untargeted: the robot is broken, not steered toward an attacker-chosen action, which blunts its value for precise hijacking. The authors contrast their "attack one step" finding with the training-time backdoor FlowHijack, which needs a wide early window, but that is a different threat model, so the clean reversal is only as convincing as you find the comparison.