How FSDP shards layers across GPUs in a HuggingFace Jobs GRPO setup
FerdinandMom · x · 2026-11-18
While testing AsyncGRPO + LoRA on Qwen3-30B-A3B with HuggingFace Jobs (3 vLLM Jobs + 2 trainer Jobs), the author explains how default FSDP works: each GPU stores only 1/4 of every layer, GPUs all-gather the full layer before computing it, then drop the borrowed pieces. GPUs talk over NVLink inside a Job, but only TCP between Jobs — setting up the thread's optimization story.
Related event: HSDP Beats FSDP by 40% in Async GRPO Training on HuggingFace Jobs(5 posts)→
More from Infra
- Voice agent team cuts P95 time-to-first-voice by 50% via single-datacenter migration — itsOmSarraf_ · 2026-11-18
- Satnam meets Prof. George Constantinides in Cambridge on UK AI infrastructure — satnam6502 · 2026-11-18
- Night aerial photo shows SpaceX Macrohard's Nvidia-powered datacenter campus ablaze with lights — PaulYacoubian · 2026-11-18
- TomorrowX Summit: how credit, power and compute traders underwrite the AI buildout differently — BenBajarin · 2026-11-18
- Nvidia's $100bn+ AI investment web under scrutiny as $500bn finance platform lands zero deals — GaryMarcus · 2026-11-18
- HuggingFace Jobs GRPO experiment: one code change makes each training step 40% faster — FerdinandMom · 2026-11-18