Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication
Yijun Ma, Siyuan Shen, Tiancheng Chen, Akhil Langer, Jiri Kraus, Benjamin Glick, Craig Belusar, Jeff Hammond, Torsten Hoefler
cs.DC
2026-06-04
A source-level study of NVSHMEM 3.3.9 finds 313 GB/s intra-node put, while inter-node AllReduce stays below 0.2 GB/s versus NCCL NVLS Tree at 252 GB/s.
NCCL made multi-GPU collectives the default, but its conventional interface is still host-launched. Expert parallelism, sparse all-to-all, and halo exchange are data-dependent and fine-grained; a CPU in the control path burns latency. NVSHMEM takes a different route. It ports the OpenSHMEM PGAS model onto GPUs so a kernel can put, get, and atomize remote symmetric memory itself.
How that actually works is scattered across docs, source, and war stories. ETH Zürich and NVIDIA walk NVSHMEM 3.3.9 from the programming model down to transports, then use DeepSeek's DeepEP as a production case. The goal is to mark where the library approaches the hardware ceiling, and where the built-in collectives currently fail.
The symmetric heap is the foundation. At init, CUDA VMM reserves a virtual range; physical pages commit on demand. Heaps of P2P-reachable GPUs are mapped at fixed offsets, so a remote address is peer-heap-base plus the local offset, and a kernel RMA is a load or store. Off-node or non-P2P peers take the slow path: IBGDA lets the GPU post RDMA to the NIC; otherwise a descriptor lands in host-pinned memory and a CPU proxy issues the op.
Collectives sit on that RMA layer. Each team owns a pSync buffer. Small messages use LL and LL128, packing payload and arrival flags into one atomic write. Algorithm choice is a rule tree, not an analytical model. Device collectives are scoped to a thread, warp, or block; there is no public grid variant. Multi-CTA execution exists only on the host on-stream path, and only when NVLS is available, by duplicating teams.
DeepEP's training path splits the cluster into eight NVSHMEM worlds so cross-node RDMA runs only between GPUs with the same local slot, then reshuffles over NVLink. The inference path drops the NVLink hop and uses IBGDA puts plus atomics. NVSHMEM is the inter-node substrate; the pipeline is DeepEP's.
Numbers come from a CoreWeave H200 cluster, NVLink-4 inside the node and ConnectX-7 between nodes. Device RMA:
| Path | Op | Peak bandwidth |
| Intra-node P2P | bulk put | 313 GB/s |
| Intra-node P2P | bulk get | 141 GB/s |
| Intra-node P2P | scalar p | 172 GB/s |
| Intra-node P2P | scalar g | below 9 GB/s |
| Inter-node IBGDA | bulk put / get | 48.0 / 48.2 GB/s |
| Inter-node IBGDA | scalar p / g | 15.6 / 1.28 GB/s |
Intra-node put still sits short of the 450 GB/s unidirectional NVLink reference. Scalar get cannot pipeline: each remote load waits for the value. Intra-node latency is about 1.3–2.5 μs; IBGDA bulk at 256 B is about 9.4–9.5 μs.
AllReduce is the sharper split. The host on-stream multi-CTA path reaches 264 GB/s, near NCCL NVLS at 276 GB/s. A forced single-CTA device-block path peaks at 30 GB/s. Across nodes both NVSHMEM variants stay below 0.20 GB/s, while NCCL ring hits 180 GB/s and NVLS Tree 252 GB/s. Small-message device latency inside a node is 3.8–7.1 μs, competitive with NCCL; by 64 KiB across nodes NVSHMEM is in milliseconds and NCCL remains in tens of microseconds.
NVIDIA's GIN paper already compared NCCL GIN with NVSHMEM on DeepEP; dispatch and combine usually land within 1–2%. This study does not rerun that experiment.
For MoE communication and custom kernels, NVSHMEM is still the low-level, GPU-initiated RMA substrate. NCCL's device API and GIN are absorbing symmetric memory and device-initiated ops into a collective library, and the GIN paper grants that NVSHMEM still wins as a one-sided substrate. Built-in collectives, especially inter-node AllReduce, should not be the reason to pick NVSHMEM today.
This is systems anatomy, not a new algorithm. For teams already on NVSHMEM, the value is a single map of fast versus slow paths, team constraints, and collective gaps.
The authors flag weak multi-CTA support in built-in collectives and almost no inter-node collective tuning. The microbenchmarks are illustrative, and some inter-node NVSHMEM AllReduce points timed out. DeepEP coverage stops at V1; V2 already moved to NCCL GIN. The analysis targets 3.3.9, with public APIs checked against 3.5.19, so internals may shift. There is no head-to-head with NCCL GIN on the same microbenchmarks.