One-line fix cuts training step time 40%: HSDP keeps all-gathers on NVLink

FerdinandMom · x · 2026-11-18

Wrap-up of the cross-node LoRA training optimization thread: 40% less time per step and 1.67x more tokens, by switching to HSDP so all-gathers stay on NVLink (330 GB/s) while TCP only carries LoRA gradients and samples. Lesson: look at your network topology.

Related event: Slow Cross-Node LoRA Training Traced to TCP Bottleneck; HSDP Yields 1.67x Throughput(2 posts)→

Original post →

More from coding & agent

coding & agent channel →