Stas Bekman open-sources torch-dist-bench: PyTorch-only NCCL collective benchmarking
StasBekman · x · 2026-10-15
ML engineer Stas Bekman released torch-dist-bench, an nccl-tests-like benchmark that measures real bandwidth and latency of all torch.distributed collectives (allreduce, allgather, reducescatter, alltoall, etc.) across payload sizes, with tables and plots.
- Only requires PyTorch to run — suited for apps calling NCCL collectives via torch.distributed rather than hand-written CUDA kernels
- Successor to all-reduce-bench, hosted in his ml-engineering repo
- Optional --with-host-overhead includes PyTorch's per-call host overhead; sample B200 8-GPU plots show large bandwidth gaps between collectives at small message sizes
- Use case: compare reality against advertised network specs
More from Infra
- Dallas Fed: AI data center debt issuance equals ~1/8 of Treasury duration supply — _onionesque · 2026-10-16
- Anthropic ditched Git as agents pushed merges to 30,000 a day, up 10x in 4 months — shashib · 2026-10-16
- NVIDIA ships two Nemotron-3.5 open models in one day: Super for H100, Bolt for RTX — kuchaev · 2026-10-16
- Anthropic now merges 30,000 code changes a day, reviews code by 'intent' not lines — Miles_Brundage · 2026-10-16
- TSMC guides ~$176B revenue for 2026, audits customer buildouts and power before expanding — BenBajarin · 2026-10-16
- Physion Labs unveils NexTron: up to 20.9x SFT throughput vs DeepSpeed on H100 — du_yilun · 2026-10-16