VLA fine-tuning in bf16 with no fp32 master copy erases 79% of updates, costing up to 75 points

Full-Weight VLA Fine-Tuning Without an fp32 Master Copy Silently Erases Most Updates

Shubham Singh

cs.RO

2026-10-23

9 of 12 open VLA codebases fine-tune with bf16 weights and no fp32 master copy, so AdamW rounds most updates away; adding the copy gains up to 75.8 points on LIBERO-Object.

What problem this solves

Fine-tuning a VLA (a vision-language-action model that maps camera images and language instructions straight to robot actions) is routine now, with LeRobot, openpi, OpenVLA, and RLinf each offering a ready entry point. To save memory, several of them cast the trainable weights to bf16 (a 16-bit float with only 8 significant bits) before building the optimizer. The trouble comes next: PyTorch's AdamW then keeps its moments in bf16 and adds every update directly into the bf16 weights. Proper mixed-precision training keeps an fp32 master copy of the weights and rounds only the copy used for compute; without one, any update smaller than about 1/512 of a weight rounds back to nothing.

Fine-tuning learning rates sit at 1e-5 to 1e-4 and most AdamW steps are smaller still, so a large share of updates disappears without an error while the loss keeps falling. This paper measures the damage, audits the ecosystem, and shows the fix costs one line.

Method

Three layers of evidence, from arithmetic to live training.

Then two audits. Of 12 open VLA codebases checked at pinned commits, 9 ship a default or documented recipe that stores trainable weights in bf16 with no copy; 4 of them train full weight matrices, with bounds from 12% (LeRobot's SmolVLA) to 94% (RLinf's PPO recipe). GR00T, CogACT, openpi's JAX trainer, and OpenVLA's full fine-tuning are not affected. A runtime check caught a trap that reading the code missed: LeRobot's SmolVLA default looks safe but keeps 96.6M of its 99.9M trainable weights in bf16, because the action expert is built from a VLM config that carries bf16 and --policy.dtype=float32 is silently ignored. LeRobot's π0/π0.5 configs default to fp32, but the official fine-tuning guides add --policy.dtype=bfloat16, dropping 3.61B trainable weights into bf16.

The Hub audit reads file headers without downloading models. Of 300 sampled SmolVLA fine-tunes, the 289 that store a full-weight action expert are 99.3% bf16. Of 300 recent π0.5 fine-tunes with a training configuration, 84% set the bf16 flag. In 40 of the bf16 fine-tunes, a median 22% of the elements in six expert weight matrices are still bit-identical to the base after training; a from-scratch control gives 0.1%.

Results

The head-to-head runs on LIBERO-Object vary only how the trainable weights are stored, score 500 fixed initial states, and follow a rule fixed before any run: a gap counts as large only if it clears 10 points and the seed spread.

Model and settingbf16, no copyfp32 copyGap
SmolVLA, lr 1e-4 (default), open-loop 50-action chunks59.3%69.5%+10.2
SmolVLA, lr 1e-54.3%49.3%+45.1
SmolVLA, lr 1e-4, re-planning every 10 actions88.4%92.6%+4.2
π0.5, lr 2.5e-5 (default)89.0%97.7%+8.7
π0.5, lr 2.5e-65.1%80.9%+75.8

The table compares against LeRobot's own recipe; two more comparisons against all-bf16 storage give +14.3 and +49.2. In all seven, every run with the copy beats every run without it (paired McNemar, p as low as below 1e-12), and all gaps except +8.7 and +4.2 clear the 10-point line. The probe measures 79–99% of updates erased at lr 1e-4 and 96–99.95% at 1e-5; with the copy, at most 0.2% of elements stay unchanged per step.

The loss curve shows nothing. At the default rate the final losses are 0.30 and 0.29 against 0.26 while success differs by 10–14 points; at the lower rate the no-copy loss still drifts down. Both low-rate settings are undertrained, with even the copy scoring 17 and 20 points below its own default-rate result. Total rented compute came to about $110.

Why it matters

If you fine-tune VLAs from the official guides, run one check first: {p.dtype for p in model.parameters() if p.requiresgrad} showing torch.bfloat16 means you are affected. The fix keeps trainable weights in fp32 and computes in bf16 under autocast: a one-line dtype change for SmolVLA in LeRobot (0.8 GB more peak memory), and for π0/π0.5 the flags --policy.dtype=float32 plus --accelerator.mixedprecision=bf16; π0.5's fp32 weights and AdamW states fit on one 80 GB GPU at batch 32 with gradient checkpointing (about 75 GB peak). Diagnosis is cheap too, since E(η) computes from released weights in minutes. Two cautions: the copy makes each step effectively larger, so a learning rate tuned under bf16 may need to come down, and two single-seed RL reports saw collapse at the unchanged rate; LoRA adapters at their default 2–5e-4 rates keep their updates and are largely unaffected.

Limitations

The author's own list is thorough: two models, one LIBERO suite, two to three seeds per setting, shortened schedules, and SmolVLA scored mostly on open-loop 50-action chunks; no ablation between the erased updates and the bf16 optimizer moments; other VLAs, tasks, and schedules untested. The scorer passes its check for π0.5 but not fully for SmolVLA (LeRobot's published checkpoint scores 67.6% with chunks against a required 88%, recovering to 93.4% when re-planning, so the shortfall traces to the chunks, not the simulator), so SmolVLA's success rates compare settings only within this paper. Pre-registration was a dated personal log rather than an outside registry, and some tests were added after scoring, which the paper discloses.

Keep the headline numbers in proportion. The two biggest gaps (+75.8 and +45.1) come from deliberately undertrained runs at a ten-times-lower learning rate; at default rates the gaps are +10.2 and +8.7, one just over the line and one just under. For anyone following the guides at default rates, the closest number is π0.5's +8.7, which removes 79% of its failures. Trained checkpoints were not kept; code and logs are available from the author on request.

Terms

Source

What people are saying

Related papers

All paper explainers