New GPU collective paper cuts LLM inference latency to within 7% of the speed-of-light bound
algo_diver · x · 2026-07-21
Key point
A new paper, “Every μs Matters: Achieving Near Speed-of-Light Latency in GPU Collectives,” targets one of the hidden bottlenecks in long-context LLM inference: collective communication latency.
What the paper says
- The authors argue that modern inference workloads are increasingly latency-bound, not bandwidth-bound, because long-context decoding triggers many small collective ops.
- They identify three main levers for near-optimal design:
- synchronization-free execution
- efficient use of symmetric memory
- multicast support
- Using NCCL’s device-side API, they build custom collective kernels and then integrate them with NCCL microbenchmarks.
- The result: substantial latency reductions for short and medium messages, with overhead reduced to within 7% of the absolute speed-of-light lower bound.
Why it matters
- The paper reports improved inter-token latency and throughput for LLM inference.
- It also improves cuSOLVERMp, showing benefits beyond LLMs into broader HPC workloads.
More from Infra
- NVIDIA starts rolling out 102.4 Tbps Spectrum-6 switches for Vera Rubin AI factories — nvidia · 2026-07-21
- Mustafa Suleyman says Microsoft is preparing for an OpenAI exit, while a new chip costs 30% less than GB200 — thoefler · 2026-07-21
- Speculative decoding boosts Qwen3.6-27B on one 5090, but slows crowded servers — luke_pacman · 2026-07-21
- NVIDIA says Blackwell Ultra hit 1,648 TFLOPs per GPU on DeepSeek-V3 671B training — NVIDIAAI · 2026-07-21
- A shopping app demo ties OpenTelemetry, Dynatrace and Port into agentic ops — Pavan_Belagatti · 2026-07-21
- Nvidia Rubin is coming, pointing to the next AI compute platform — ezyang · 2026-07-21