The Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models
Chen Xu, Rishi Shah, Hadas Kress-Gazit, Haruki Nishimura, Masha Itkina
cs.RO, cs.AI
2026-09-23
Non-Gaussian action priors closer to the target do not beat a Gaussian when fine-tuning LBM 1.0, π0.5, and GR00T N1.5, except possibly at ~20 demos per task.
Diffusion and flow-matching policies generate robot actions by transporting samples from a prior onto an observation-conditioned target. When a single-task policy is trained from scratch, swapping the usual Gaussian for a prior that sits closer to the target has already raised success rates on LIBERO, MetaWorld, and Adroit.
Large Behavior Models take a different route: pretrain on diverse vision, language, and robot data, then fine-tune on a modest set of task demos. LBM 1.0, π0.5, and GR00T N1.5 all default to an isotropic Gaussian. Flow matching does not require that choice, so the prior can be swapped with no change to the training loop. The hope is short transport and easier fine-tuning. Toyota Research Institute, Woven by Toyota, and Cornell put that hope on pretrained LBMs and got a negative result.
Only the prior changes. Task, data, architecture, optimizer, and fine-tuning steps stay locked. Seven priors: the default Gaussian Z; three constructions that reuse a frozen pretrained LBM; and three literature baselines, Cocos (a condition-dependent Gaussian), BRIDGER (a CVAE), and k-NN retrieval. The three in-house priors are Apre, an action chunk taken from the frozen policy at half the usual denoising budget; Apre plus Gaussian jitter in action space; and Gaussian jitter in encoder embedding space before the frozen policy produces a0.
Four settings cover the design space. Setting A first fine-tunes LBM 1.0 with a Gaussian on 50% of the demos, then treats that checkpoint as the pretrained model for a second round, so a weak zero-shot prior cannot explain a null. Setting Bsim follows the usual deployment recipe on eight LBM Eval tasks held out from pretraining, sweeping 5%, 25%, 50%, 75%, and 100% of the demos. Setting Breal runs blinded A/B tests on a bimanual Franka FR3. Setting C repeats the comparison for π0.5 and GR00T N1.5 on 34 RoboCasa tasks. Tests use the STEP framework with Bonferroni correction; compact letter display marks which methods are separable.
From scratch, priors help. In Setting A, Apre reaches about 28% mean success against 18% for the Gaussian, a statistically separable gap. Apre is also about 3x closer to ground-truth actions in ℓ2.
The same priors lose that edge under fine-tuning. After the second-round fine-tune in Setting A, none of the seven is separable from the Gaussian, and Apre is empirically the worst. In Bsim, from 25% data upward, most priors share one letter group. The exception is the 5% slice, roughly 10-20 demos per task: the embedding-jitter prior hits 25.9% success against 18.9% for the Gaussian. On hardware, 1250 rollouts put four priors in the same group with partial-progress scores of 71-77%; only BRIDGER, whose CVAE training was unstable, falls to about 56%. On RoboCasa, the Gaussian and Apre-plus-action-noise tie at the top, while pure Apre and retrieval lag, even though Apre remains about 3x closer in ℓ2.
Diagnostics point at the encoder. LogME feature quality after fine-tuning is inseparable across priors. Open-loop action ℓ2 sits in 5.7-6.4 and cosine means in about 0.79-0.83; that gap does not show up in closed-loop success. Embeddings do diverge: cosine to the pretrained encoder is about 0.84 for the Gaussian and 0.55-0.64 for Apre. Cutting the encoder learning rate to one-tenth of the action head drops simulation success from 56-59% to 41-43%, and hardware Gaussian success from 37% to 28%. Inside each learning-rate band, swapping the prior barely moves the needle.
For fine-tuning an LBM with a flow-matching action head, a standard Gaussian is enough. Extra engineering for a closer prior does not buy a measurable gain. The budget belongs on whether the encoder is actually being trained. From-scratch single-task policies are outside this negative result; the published non-Gaussian priors still hold there. The 5% slice is a hint that embedding-space jitter can supply an inductive bias the Gaussian cannot when demos are scarce. The boundary is not mapped.
The claim covers three LBMs. Much larger or smaller models, or unusual pretraining mixtures, are untested. Fine-tuning sets are about 20-400 demos per task; whether the null survives much larger or more diverse data is open. The story that the pretrained encoder dominates the prior is plausible at other scales and still unverified. LogME, ℓ2, and cosine are standard transfer probes; LEEP and H-Score were not run. BRIDGER's CVAE hit NaN losses on both sim and hardware, and only stable seeds entered the eval. π0.5 and GR00T were pretrained mostly on real data, so Apre and retrieval, which lean entirely on a frozen copy, take the domain gap more directly than noise-regularized priors.