Deep Dive: TPU and GPU Cluster Communication
thehiphopswami · x · 2026-07-15
This is an in-depth blog post about TPU/GPU clusters and collective communication, focusing on the underlying mechanics of scaling training and inference.
Topics covered include:
- TPU cluster topology: superpods, slices, DCN, PCIe, ICI
- How All-Gather, Reduce-Scatter, and All-Reduce work
- The role of All-to-All in MoE token dispatch
- NVIDIA GPU cluster topologies and related communication paths
The author's goal is to demystify these seemingly abstract communication primitives, helping readers truly grasp the foundations of scaling MoE and dense transformer models.
Related event: Deep Dive on Collective Communication in TPU/GPU Clusters Gains Traction(5 posts)→
More from Infra
- AWS tutorial: fleet-wide edge CV model lifecycle with SageMaker SDK v3, ~$0.07 per run — AWS ML Blog · 2026-10-27
- Qwen3.8-27B hits 40 tok/s on Jetson Thor with INT4 + MTP, accuracy beats FP16 — No-Dragonfly6246 · 2026-10-27
- Celesto launches CI runners nearly 90% cheaper than GitHub Actions, with agent sandboxes — aniketmaurya · 2026-10-27
- Supabase founders on rebuilding the backend for a billion AI-built apps — Latent Space · 2026-10-27
- AW-0 runs at 230ms on a single RTX 5090, 2.4x faster than autoregressive code planning — trevordarrell · 2026-10-27
- Site tracks US city council topics: data centers rank only behind pickleball courts — zakkohane · 2026-10-27