Google's Token Volume Grew 7x to 3.2 Quadrillion a Month — How Cerebras and Gimlet Keep Inference Up
AI Engineer · youtube · 2026-10-12
At AI Engineer World's Fair, Daniel Kim of Cerebras and Gimlet Labs co-founder Michelle explained how inference keeps getting faster and cheaper, against the backdrop of Google's token volume growing from 480 trillion to 3.2 quadrillion tokens per month.
- Runtime level: decoding is memory-bound and the KV cache balloons with context; fixes include splitting models across GPUs, continuous batching, paged KV cache, prompt caching, and chunked prefill.
- System level: prefill/decode disaggregation, speculative decoding, and attention/FFN disaggregation.
- Gimlet demoed its multi-silicon inference cloud and an agent that writes optimized kernels, extending to agents for silicon, compilers, and ops.
More from Infra
- 'AI always maxes out your GPU'? Devs push back on the myth — mgostIH · 2026-10-12
- Yandex reports data center damage causing service outages, 'Internyet' meme ensues — kingslyj · 2026-10-12
- Linus Torvalds is using Claude Code on a hefty Linux kernel performance improvement — PointmanW · 2026-10-12
- 4070 Ti Super beats RTX 5070 2x on MiniMax H3 int8 in local inference benchmark — smellyelon · 2026-10-12
- Strata Runs a Big Open Model Locally on a Normal Gaming PC — alex_verem · 2026-10-12
- Why keep making LLMs bigger? A case for DeepSeek and Qwen's small-model extremism — gnukeith · 2026-10-12