Slow Cross-Node LoRA Training Traced to TCP Bottleneck; HSDP Yields 1.67x Throughput
Cross-node LoRA training was bottlenecked by FSDP forcing a 61GB model through 15GB/s TCP links; switching to HSDP to keep all-gather on NVLink cut per-step time by 40% and boosted token throughput 1.67x.
2026-11-18 ~ 2026-11-18 · 2 related posts
- The culprit: 61GB frozen base all-gathered over 15GB/s TCP every micro-batch — FerdinandMom · 2026-11-18
- One-line fix cuts training step time 40%: HSDP keeps all-gathers on NVLink — FerdinandMom · 2026-11-18