HuggingFace Jobs GRPO experiment: one code change makes each training step 40% faster
FerdinandMom · x · 2026-11-18
Testing AsyncGRPO + LoRA on Qwen3-30B-A3B with HuggingFace Jobs (3 vLLM Jobs + 2 trainer Jobs), the author found GPUs talk over NVLink inside a Job but only TCP between Jobs. Same GPUs, same model — one simple code change cut 40% off each training step. The thread breaks down the cause and the fix.
Related event: HSDP Beats FSDP by 40% in Async GRPO Training on HuggingFace Jobs(3 posts)→
More from Infra
- Nvidia's $100bn+ AI investment web under scrutiny as $500bn finance platform lands zero deals — GaryMarcus · 2026-11-18
- The culprit: 61GB frozen base all-gathered over 15GB/s TCP every micro-batch — FerdinandMom · 2026-11-18
- A trillion-dollar pile: photo rounds up Nvidia's entire GPU lineup — LinusEkenstam · 2026-11-18
- Practical deep dive: when software engineers should drop frontier lab APIs — mikeflache · 2026-11-18
- Starlink at 11 years: 11,100+ satellites in orbit, 13.6M subscribers across 174 markets — XFreeze · 2026-11-18
- Ex-Nvidia sales chief Jay Puri drawn into chip-diversion probes, may be interviewed — steph_palazzolo · 2026-11-18