One-line fix cuts training step time 40%: HSDP keeps all-gathers on NVLink
FerdinandMom · x · 2026-11-18
Wrap-up of the cross-node LoRA training optimization thread: 40% less time per step and 1.67x more tokens, by switching to HSDP so all-gathers stay on NVLink (330 GB/s) while TCP only carries LoRA gradients and samples. Lesson: look at your network topology.
More from coding & agent
- Managing four Docker hosts with Komodo and GitHub in one dashboard — ruhanirabin · 2026-11-18
- Lily Ray's workflow: use ChatGPT to compile your past work into shareable links — lilyraynyc · 2026-11-18
- Drawing humanity's history with JavaScript: cave art to scientific revolution — prasenx · 2026-11-18
- After two years on Codex and Claude, dev says you simply can't quit — rachittshah · 2026-11-18
- Cursor teases Origin, 'a git forge for the agentic era' — himanshustwts · 2026-11-18
- Sysdig CISO: LLMs can rewrite the harness rules meant to govern them — massimosgrelli · 2026-11-18