DeepMind's guide to sharded matrix multiplication: the math behind training LLMs on 10k TPUs
zacharynado · x · 2026-09-16
Google DeepMind researchers (Jacob Austin, Roy Frostig, Reiner Pope et al.) published part 3 of 'How To Scale Your Model': a systematic theory of sharded matrix multiplication built on the cost of TPU communication primitives. It explains how to partition LLM parameters across thousands of accelerators efficiently, covering collective operations, partitioning notation, and why training and inference scale to larger topologies for different reasons (step time vs latency).
More from Infra
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02