Deep Dive: TPU and GPU Cluster Communication
thehiphopswami · x · 2026-07-15
This is an in-depth blog post about TPU/GPU clusters and collective communication, focusing on the underlying mechanics of scaling training and inference.
Topics covered include:
- TPU cluster topology: superpods, slices, DCN, PCIe, ICI
- How All-Gather, Reduce-Scatter, and All-Reduce work
- The role of All-to-All in MoE token dispatch
- NVIDIA GPU cluster topologies and related communication paths
The author's goal is to demystify these seemingly abstract communication primitives, helping readers truly grasp the foundations of scaling MoE and dense transformer models.
Related event: Deep Dive on Collective Communication in TPU/GPU Clusters Gains Traction(5 posts)→
More from Infra
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27
- NVIDIA Claims HGX Rubin NVL8 Delivers 10x More Tokens per Megawatt Than B200 — thursdai_pod · 2026-10-27
- Together lets multiple LoRA post-training experiments share the same GPU capacity — togethercompute · 2026-10-27
- Commoditization dilemma: either human labor or GPUs become the commodity — _AustinCalvert_ · 2026-10-27
- Anthropic S-1: $112B Cash, $80B "Loss" Is a $72B Non-Cash Charge — robleclerc · 2026-10-27