Hugging Face TRL Gets Major Speedup: AsyncGRPO Training Steps Up to 3.5x Faster
ben_burtenshaw · x · 2026-09-23
A major optimization for AsyncGRPO in Hugging Face's TRL is making training steps up to 3.5x faster. The insight: in GRPO the same prompt is sampled G times, and multi-turn rollouts share most of their conversation history, with only the suffix changing as tool results arrive. By leveraging message-level tokenization to share long prefixes across rollouts and using sample packing that balances attention cost across DP ranks, redundant computation in the rollout phase is drastically reduced.
More from Infra
- NVIDIA reportedly cancels RTX 5080 24GB over GDDR7 supply and cost — ceciletamura · 2026-11-11
- Rivet launches SQLite Forking: 20ms agent-thread forks with zero data copying — samgoodwin89 · 2026-11-11
- French netizen proposes datacenters pay local residents compensation — clemnt · 2026-11-11
- GPU Rental Prices Hit Record Highs: B200 at $6.97/hr, H100 at $3.06 — inductionheads · 2026-11-11
- A Datacenter in Every Office: Building Work-Safe Local AI Racks — dee_hw · 2026-11-11
- "Compute is about power": the AI race's real bottleneck is energy, not chips — _sholtodouglas · 2026-11-11