Mixed-precision, mixed-accelerator samplers cut RL training cost with off-policy correction
xiuyu_l · x · 2026-10-14
nanjiangwill (with Modal) argues that a sampler on a different precision (MXFP8, MXFP4, NVFP4) or accelerator is just an off-policy sampler—like stale samples in fully async RL—and off-policy correction doesn't care why the sampler differs.
Key points:
- RL training typically locks into homogeneous clusters of the latest accelerators for numerical stability, discarding inference optimizations for other vendors/generations and inflating cost.
- They trained a model on agentic coding tasks with fully asynchronous RL using mixed-precision, mixed-accelerator samplers; with off-policy correction, training stayed stable through convergence, matching homogeneous samplers at lower cost.
- Thesis: sampler and trainer are separate systems built for different purposes; diversifying inference hardware means idle compute could be repurposed for RL rollouts, enabling cleaner system designs and new algorithms.
A follow-up post on the infra side is teased.
More from Infra
- SemiAnalysis teases TPUv7 TCO data: $0.181/M tokens vs $0.222 B200 — firstadopter · 2026-10-14
- Serverless tool calling could slash agent costs 30x by running 1000 agents per sandbox — madhavsinghal_ · 2026-10-14
- Prompt caching kills easy/hard model routing: cached big model now cheaper than small one — Lance_Saul_85 · 2026-10-14
- BofA lifts global memory TAM forecast to $2T by 2030, run rate already over $1T — Beth_Kindig · 2026-10-14
- Ayar Labs, Wiwynn and GUC demo integrated optical compute system with live telemetry at OCP — bookwormengr · 2026-10-14
- AnythingLLM ditches Ollama for Lemonade, shrinking engine from 1.8GB to 137MB — jfowers_amd · 2026-10-14