How FSDP shards layers across GPUs in a HuggingFace Jobs GRPO setup

FerdinandMom · x · 2026-11-18

While testing AsyncGRPO + LoRA on Qwen3-30B-A3B with HuggingFace Jobs (3 vLLM Jobs + 2 trainer Jobs), the author explains how default FSDP works: each GPU stores only 1/4 of every layer, GPUs all-gather the full layer before computing it, then drop the borrowed pieces. GPUs talk over NVLink inside a Job, but only TCP between Jobs — setting up the thread's optimization story.

Related event: HSDP Beats FSDP by 40% in Async GRPO Training on HuggingFace Jobs(5 posts)→

Original post →

More from Infra

Infra channel →