Google DeepMind Publishes Free Book on Scaling LLMs Across TPUs and GPUs
goyal__pramod · x · 2026-09-05
Google DeepMind researchers (Jacob Austin, Sholto Douglas, Roy Frostig, and others) released the free online book How To Scale Your Model in February 2025, demystifying how LLMs run and scale on TPUs (and GPUs).
- Coverage: how accelerators work and communicate, Roofline performance models, parallelism schemes for training and inference (AllGather and other collectives), and how to estimate training cost and memory needs.
- Practical aims: gauge how close your model is to hardware optimum, pick parallelism strategies at any scale, and size hardware for self-hosting.
- Premise: performance optimization needn't be alchemy — simple principles apply from one accelerator to tens of thousands.
- Prerequisites: Transformer/LLM training basics and ideally some JAX familiarity.
More from Infra
- AMD, Cisco and Saudi Arabia's HUMAIN deploy MI335X GPUs, planning up to 250MW — Beth_Kindig · 2026-09-05
- Reef launches inference-native infra that serves self-improving agents without downtime — pliang279 · 2026-09-05
- From 1D to 2D int8 kernels: a hands-on GPU internals learning path — goyal__pramod · 2026-09-05
- Prime Super Flash MoE: 1.2x BF16 and 1.6x MXFP8 speedups over upstream on B200 — retr0jirachi · 2026-09-05
- How to Estimate tokens/sec on Your Hardware: The VRAM Bandwidth Formula — Pyrolistical · 2026-09-05
- Google Cloud proposes Cloud Run sandboxes to fix coding agents' repair loop — rseroter · 2026-09-05