HSDP cuts async GRPO step time 40% and boosts tokens/s 1.67x vs FSDP on Qwen3-30B
FerdinandMom · x · 2026-11-18
A hands-on benchmark of async GRPO + LoRA on Qwen3-30B-A3B via Hugging Face Jobs (TRL AsyncGRPOTrainer, 3 vLLM rollout Jobs, trainer on 4x H200 across two Jobs) shows switching FSDP to HSDP cuts time per step by 40% and raises throughput 1.67x.
- Root cause: GPUs within a Job talk over NVLink, but Jobs only over TCP; FSDP keeps shuttling weights across that slow link, while HSDP keeps the traffic local.
- The fix is a one-line change with transformers.distributed: DistributedConfig(fsdpsize=4) becomes DistributedConfig(fsdpsize=2, dpreplicatesize=2).
- Full repro code and results are open-sourced on GitHub (hf-jobs-fsdp-hsdp), including gateway/launch/train scripts.
Related event: HSDP Beats FSDP by 40% in Async GRPO Training on HuggingFace Jobs(3 posts)→
More from Infra
- Nvidia's $100bn+ AI investment web under scrutiny as $500bn finance platform lands zero deals — GaryMarcus · 2026-11-18
- The culprit: 61GB frozen base all-gathered over 15GB/s TCP every micro-batch — FerdinandMom · 2026-11-18
- HuggingFace Jobs GRPO experiment: one code change makes each training step 40% faster — FerdinandMom · 2026-11-18
- A trillion-dollar pile: photo rounds up Nvidia's entire GPU lineup — LinusEkenstam · 2026-11-18
- Practical deep dive: when software engineers should drop frontier lab APIs — mikeflache · 2026-11-18
- Starlink at 11 years: 11,100+ satellites in orbit, 13.6M subscribers across 174 markets — XFreeze · 2026-11-18