TRL reproduces Modal's mixed-precision async RL: plain GRPO collapses, SC+MIS holds 0.72
QGallouedec · x · 2026-10-16
- Modal argued that RL training can use fully asynchronous, mixed-precision, mixed-accelerator samplers with off-policy correction instead of a homogeneous accelerator cluster.
- TRL reproduced this in AsyncGRPO with Qwen-1.5B, a vLLM sampler in FP8 (weights + KV) and a bf16 trainer.
- Results: plain GRPO collapses after 150 steps (reward 0.56 → 0.10); ratio caps or masked IS alone don't save it; score centering + MIS trains to 0.72, on par with a bf16 sampler.
- Takeaway: FP8 samplers are viable for stable RL training with the right off-policy corrections, validating Modal's cheaper mixed-precision setup.
More from Infra
- Three CoreWeave co-founders have sold $3.3B in shares since IPO amid credit worries — SumitGup · 2026-10-16
- Cherry pi on a Raspberry Pi: the tiny-model rumor developers are watching — sull · 2026-10-16
- CoreWeave's debt costs falling: DDTL spreads tightened from S+9.62% to S+2.25% — BenBajarin · 2026-10-16
- SoftBank lines up US IPOs for SB Energy and AI robotics spinoff Roze, valuations top $50B — nordicinst · 2026-10-16
- Hugging Face scales Indic pretraining with 2.8T tokens and an open data pipeline — Hugging Face Blog · 2026-10-16
- NVIDIA Vera Rubin NVL72 hits up to 5.7x GB300 throughput per MW on agent benchmark — ArtificialAnlys · 2026-10-16