Hugging Face surveys 16 open-source RL libraries: async disaggregation is the consensus
Thom_Wolf · x · 2026-09-05
Hugging Face published a long-form survey comparing 16 open-source RL training libraries, distilling engineering lessons from async RL pipelines. Thom Wolf (ex-NVIDIA) shared it, noting his small team moved to Ray.
Key points:
- Problem: In synchronous RL training, rollout generation dominates wall-clock time — a batch of 32K-token rollouts on a 32B model can take hours while training GPUs sit idle
- Converged solution: Disaggregate inference and training onto separate GPU pools, connect via a rollout buffer, sync weights asynchronously
- Compared across 7 axes: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, distributed backends
- Findings: Ray dominates orchestration (8/16); NCCL broadcast is the default weight-transfer method; staleness handling ranges from dropping old samples to importance-sampling correction; LoRA support is sparse
The most systematic cross-comparison of open-source RL infrastructure to date — useful for post-training engineering teams choosing a stack.
More from Infra
- GeneralistAI collects 500,000+ hours of real robot data, launches onchain data bounties via Robinhood — broodsugar · 2026-09-05
- NInfer vs llama.cpp vs vLLM: rigorous quality and speed test of Qwen3.8-27B NVFP4 on one RTX 5090 — bengizmoed · 2026-09-05
- Omagrid P2P compute network goes live, sharing DeepSeek and Qwen across Omarchy machines — dee_hw · 2026-09-05
- Running a 180B MoE (5B active) on one RTX 3090 + 128GB RAM: full config and 15.5 tok/s benchmarks — cezarducatti · 2026-09-05
- Gave up local video rendering on an RTX 3060 for cloud — but lost the node-based modularity — Suspicious_Pizza9529 · 2026-09-05
- Anthropic IPO investors press for revenue per token and per gigawatt metrics — Hesamation · 2026-09-05