Berkeley's QF3 makes RL work for flow policies on real robots with one clipped gradient tweak
andrew_n_carr · x · 2026-10-13
QF3 from UC Berkeley and Amazon FAR stabilizes off-policy RL for flow policies with a single actor-update change: clip the policy's velocity around replay actions and mask critic gradients where the critic has no data.
- Why it matters: Flow policies power many robots today, but following the critic's gradient naively makes policies chase overestimated rewards and collapse training.
- Results: LoRA fine-tuning of the ABC-VLA bimanual policy finishes bottles-in-bin 1.8× and plates-in-dish-rack 2.1× faster than the base policy; full fine-tuning on robomimic Square, Tool Hang, Transport; humanoids trained from scratch hit motion-tracking parity with FPO++ about 10× faster in wall-clock time on a single GPU.
- The same update fine-tunes Stable Diffusion 3.5 Medium using PickScore as the critic.
- Code is "coming soon"; project page and paper (arXiv 2610.08789) are live. Authors include Ken Goldberg, Pieter Abbeel, and Angjoo Kanazawa.
Related event: UC Berkeley's QF3 Enables Stable RL Fine-Tuning of Flow Policies(2 posts)→
More from Embodied
- Perseverance rover breaks off-world driving record at 28.10 miles — NASA · 2026-10-13
- ASCII Depth-Perception Art With a Mac and RealSense D435 Camera — chrismatthieu · 2026-10-13
- SFGate slams Zuckerberg's new Muse gadget: 'a parasite, don't fall for it' — sfgate · 2026-10-13
- Leaked NVIDIA slides reveal Factory AI blueprint with Nemoclaw agent framework — firstadopter · 2026-10-13
- Google DeepMind Robotics is hiring a pretraining & data quality research scientist in London — bousmalis · 2026-10-13
- SafeDriveVLA (CoRL 2026): world-model dreaming catches unsafe driving instructions before acting — rsasaki0109 · 2026-10-13