One-line HSDP tweak speeds up cross-node GRPO training by 40% on HuggingFace Jobs
Blogger FerdinandMom ran TRL's AsyncGRPOTrainer with LoRA on HuggingFace Jobs to train Qwen3-30B-A3B (3 vLLM Jobs generating rollouts, 2 trainer Jobs training across nodes), and found that a single code change cut per-step training time by 40% and boosted token throughput 1.67x — the key was rethinking the communication topology rather than the algorithm itself.
Confirmed
- Training setup: AsyncGRPO + LoRA, with 3 vLLM Jobs for generation and 2 trainer Jobs running across nodes; GPUs within a Job connect via NVLink, while Jobs only have TCP between them, with measured bandwidth topping out around 15GB/s.
- Why it was slow: under default FSDP, each GPU stores only 1/4 of every layer and must all-gather the full layer before compute; this meant the 61GB frozen base model was repeatedly transferred across Jobs on every micro-batch, bottlenecked by the slow TCP link.
- The fix: switch to HSDP, keeping all-gather communication on NVLink within each Job, so cross-Job TCP only carries lightweight LoRA gradients and samples.
- Results: 40% less time per step and 1.67x higher token throughput.
Why it matters
- This case shows that in multi-node / multi-Job distributed training, communication topology (NVLink vs TCP, which tensors go over which link) can matter more than algorithm- or framework-level tuning — a one-line config change delivered major gains.
- It's directly relevant for users fine-tuning large base models with LoRA on platforms like HuggingFace Jobs, where resources are partitioned per Job and cross-Job bandwidth is limited.
2026-11-18 ~ 2026-11-18 · 5 related posts
Primary sources
- [source] HuggingFace Jobs GRPO experiment: one code change makes each training step 40% faster — FerdinandMom · 2026-11-18
- How FSDP shards layers across GPUs in a HuggingFace Jobs GRPO setup — FerdinandMom · 2026-11-18
- The culprit: 61GB frozen base all-gathered over 15GB/s TCP every micro-batch — FerdinandMom · 2026-11-18
- [source] One-line fix cuts training step time 40%: HSDP keeps all-gathers on NVLink — FerdinandMom · 2026-11-18
- HSDP cuts async GRPO step time 40% and boosts tokens/s 1.67x vs FSDP on Qwen3-30B — FerdinandMom · 2026-11-18