Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives
Siyuan Shen, Anton Korzh, John Bachan, Tiancheng Chen, Arnav Goel, Ludwig Schneider, Pouya Kousha, Zhenhao He, Sylvain Jeaugey, Kamil Iskra, Nishank Chandawala, Jeff R. Hammond, Torsten Hoefler
cs.DC
2026-07-18
An NVIDIA team rewrites NCCL low-latency collectives to cut small-message AllReduce from 11µs to 2.4µs, near the hardware speed-of-light bound; in vLLM, tensor-parallel LLM inference gains 7-13% per-token latency.
GPU collective operations (AllReduce, AllGather, the primitives that combine data across GPUs and redistribute it) have been optimized for bandwidth for a decade. During training, gradient synchronization moves hundreds of megabytes per step; the goal is to keep the pipe full, and nobody minds a few microseconds of fixed overhead.
Long-context, small-batch inference breaks that assumption. A model that does not fit on one GPU runs under tensor parallelism (TP), which splits each layer across cards, and every generated token triggers a small AllReduce to stitch the sharded output back together. These messages are often a few kilobytes, so bandwidth is irrelevant; what bites is fixed latency. On the decode critical path, every extra microsecond becomes user-visible wait time.
The authors measure that even the fastest current implementations sit well above the hardware speed-of-light (SoL) bound, the physical floor set by the wire, the L2 cache round-trip, and memory writes. They trace the overhead to one culprit: the global memory barrier. Every synchronization step in legacy code inserts a barrier that forces all GPUs to wait for the slowest one, and that wait costs microseconds.
Four techniques, all aimed at eliminating global memory barriers.
LL (Low-Latency): pack the readiness signal into the data. Legacy code writes data, then separately signals that it is done, with a barrier in between. LL packs an 8-byte flag and 8 bytes of data into a single 16-byte atomic store; the receiver reads the flag and knows the data is there, with no separate signal step. The cost is halved effective bandwidth and doubled scratch buffer, so it suits only very small messages.
Sentinel: use the data value itself as the signal. The receive buffer is pre-filled with an unlikely sentinel value (e.g., -NaN); when real data lands, the value changes, and the receiver polls until it does. Full bandwidth, less scratch, but the buffer must be reset each round and the data must never coincidentally equal the sentinel. For larger messages and more ranks, it beats LL.
Bidirectional communication with double buffering: kill barriers between iterations. When a message must be chunked across many transfers, LL and sentinel handle each transfer but barriers remain between iterations. Bidirectional communication turns each received chunk into implicit permission to send the next one (credit-based flow control); paired with double buffering, iterations no longer need a barrier.
Two-shot LL128 atomic AllReduce: the paper core new algorithm. AllReduce is split into ReduceScatter then AllGather. In the reduce phase, threads operate in groups of eight on 128-byte cache lines; the first thread in a group sets its element to 1 as a flag, then all eight atomically add into the scratch buffer. When the flag accumulates to N (the rank count), every rank contribution has arrived. It needs only about D/N scratch space (D is the data reduced per iteration) and wastes only about 1.5% of bandwidth in FP16 (about 3% in FP32). The requirements are stiff: it needs NVLink cache-line atomic addition, supports only addition (because the embedded flag relies on commutativity), supports only FP32/FP16/BF16, and because floating-point atomic order is nondeterministic, the result is nondeterministic.
These primitives ship as a low-level ncclLLBuffer API for people writing custom kernels.
The authors first compute the SoL floor: a one-shot AllReduce lower bound is 2×(L2 round-trip) + one remote store. Measured on two GB200s, the L2 round-trip is 0.306 µs and a remote store is 0.792 µs, giving a floor of 1.404 µs.
| Setting | Baseline | New kernel | Over SoL |
| Small-message AllReduce (4 GPUs) | NCCL ring 11.0 µs | 2.37 µs | about 7% |
| 2 GPUs, 128-byte one-shot | n/a | LLBuffer | about 7% |
| 64 GPUs, multicast variant | n/a | n/a | about 70% |
At small scale the kernel nearly touches the physical floor; scaled to 64 GPUs with multicast, about 70% headroom remains.
On real workloads (vLLM, GB200):
| Metric | 4 GPUs (TP=4) | 8 GPUs (TP=8) |
| ITL (per-token latency) reduction | 7-13% | 9-11% |
| Throughput gain | comparable | comparable |
The results hold across dense (Llama-3.1-70B), mixture-of-experts (DeepSeek-V3), and hybrid-attention (Qwen3-Next) architectures. The abstract Llama-3.1-70B example on 4 GPUs: ITL down 8.7%, which maps to about 0.9% cost reduction at CoreWeave 42 USD/hour for 4 GB200. Larger models on 8 GPUs carry higher absolute cost: DeepSeek-V3 runs about 11.78 USD per million output tokens, dropping to about 10.88 USD after the optimization.
Traditional HPC is not left out. On the Alps supercomputer (Grace Hopper nodes), cuSOLVERMp distributed eigensolver improves more at m=32768, where communication is a larger share of runtime.
The key word is critical path. During training, collectives hide behind compute through overlap; during decode, every token forces a synchronization that cannot be hidden, and its latency becomes the user-visible typing speed and serving cost. Cutting that step from 11 µs to 2.4 µs is microscopic per token but real money over long outputs and large fleets.
The authors are an NVIDIA team (including Torsten Hoefler), and the implementation lands in NCCL, so this will reach most inference stacks through a library update, not stay on paper. For any team serving large models with tensor parallelism, especially long-context and reasoning (decode-heavy) workloads, this is worth tracking: once your NCCL version ships the low-latency kernels, ITL and throughput improve with no application-code change.
A cold dose of realism: a 7-13% ITL gain is real but not transformative. The value is less in any single number and more in the methodology of driving latency toward the physical floor, and the demonstration that barriers can be eliminated systematically.