New blog demystifies GPU communications, from page tables to NVLink, via SGLang internals
BanghuaZ · x · 2026-10-25
A new blog series (part I) walks through GPU communication concepts and maps each one to how SGLang uses them internally.
- Basics: pinned vs pageable host memory, NUMA placement
- Memory sharing: CPU-side sharing with SGLang usage; GPU-side IPC handle lifecycle
- Physical transport: PCIe (saturating the link), NVLink (SGLang's Custom All-reduce and V2), and NVSwitch
Future installments promise deeper coverage of VMM and symmetric memory. Useful for engineers wanting to understand how inference frameworks exploit multi-GPU communication.
More from Infra
- Hyperscaler depreciation to hit $590B by 2029, likely exceeding combined operating profits — GaryMarcus · 2026-10-25
- vLLM hits 954K tokens/s per GPU serving DeepSeek V4.1-Flash on GB300 NVL72 rack — vllm_project · 2026-10-25
- vLLM matches DeepInfer on DeepSeek-V4.1-Flash across batch sizes 8–256 — vllm_project · 2026-10-25
- UK per-capita electricity now below 1970s levels, just as AI demands power — NinaDSchick · 2026-10-25
- MCP server indexes 96k-file Linux kernel in 2m51s on a laptop — and the benchmark exposed a 1.5s query bug — Numerous_Beyond_2442 · 2026-10-25
- Hyperscaler AI capex hits $800bn, 2.4% of US GDP, dwarfing Manhattan Project — Unfair-Journalist-66 · 2026-10-25