Day 280/365 of GPU Programming: Diving into NCCL and How PyTorch Calls AllReduce
NandoDF · x · 2026-10-12
Developer levidiamode's day 279-280 of a 365-day GPU programming challenge covers NCCL (NVIDIA Collective Communications Library).
- NCCL basics: NCCL enables fast GPU-to-GPU communication with point-to-point and collective primitives — Scatter, Gather, All-to-all, AllReduce, Broadcast, Reduce, AllGather, ReduceScatter.
- PyTorch + NCCL: Following a GPU Mode lecture by Meta research scientist Daniel Johnson, the post traces how PyTorch training code ends up calling NCCL AllReduce, and why AllReduce ops run on their own CUDA stream.
- Screenshots include a training trace showing sgemm kernels interleaved with nccl::allreduce calls, plus a recommended deep-dive blog on data-parallel training.
More from Infra
- CUDA moat drives RTX Spark laptop sales; 128GB Surface variant already sold out — chemist_slime · 2026-10-12
- AWS weekly digest: Claude Haiku 5.5 lands, Nova 2.5 Sonic targets voice agents — DigitalColmer · 2026-10-12
- Heavy AI usage is driving up electricity consumption — morgymcg · 2026-10-12
- Deno open-sources celld: self-hosted distributed Durable Objects without a consensus cluster — bibryam · 2026-10-12
- Nobody knows how long a GPU lives — and your retirement is now a bet on it — LexSokolin · 2026-10-12
- FT: China rapidly builds data centers in energy-rich hinterland to power AI race — EleanorOlcott · 2026-10-12