torch-dist-bench latency plots: 9 collectives benchmarked on 8x B200
StasBekman · x · 2026-10-15
Stas Bekman shared the latency companion plots from torch-dist-bench runs on 8x NVIDIA B200: per-call median latency for 9 collectives (allreduce, allgather, gather, scatter, batchisendirecv, etc.) across message sizes from 8B to 16GiB. Small-message latencies sit in the tens of microseconds; batchisendirecv shows a notable latency plateau at mid-size messages, while bus bandwidth for most collectives climbs to 600–800+ GBps at large payloads.
More from Infra
- Anthropic ditched Git as agents pushed merges to 30,000 a day, up 10x in 4 months — shashib · 2026-10-16
- NVIDIA ships two Nemotron-3.5 open models in one day: Super for H100, Bolt for RTX — kuchaev · 2026-10-16
- Anthropic now merges 30,000 code changes a day, reviews code by 'intent' not lines — Miles_Brundage · 2026-10-16
- TSMC guides ~$176B revenue for 2026, audits customer buildouts and power before expanding — BenBajarin · 2026-10-16
- Physion Labs unveils NexTron: up to 20.9x SFT throughput vs DeepSpeed on H100 — du_yilun · 2026-10-16
- Dreamscale launches cloud inference for physical AI: 8+ robots per H100, 320ms DreamZero latency — arnie_hacker · 2026-10-16