The culprit: 61GB frozen base all-gathered over 15GB/s TCP every micro-batch

FerdinandMom · x · 2026-11-18

The author diagnoses why cross-node LoRA training is slow: with default FSDP, each GPU stores 1/4 of every layer and must all-gather the full layer before computing it; between Jobs only TCP (at best 15GB/s) is available, so the 61GB frozen base has to cross nodes on every micro-batch — the bottleneck.

Related event: Slow Cross-Node LoRA Training Traced to TCP Bottleneck; HSDP Yields 1.67x Throughput(2 posts)→

Original post →

More from Infra

Infra channel →