LLM Inference Is Bandwidth-Bound: Lemire on Why Positron Inverts GPU Design
lemire · x · 2026-09-11
Daniel Lemire argues inference — not training — is the part that must get cheap, and most hardware wasn't designed for it. LLM inference is bandwidth-bound: massive matrix-vector multiplies against barely-reused weights, closer to video streaming than simulation. GPUs were built compute-first with HBM bolted on. The fix: invert the design — stream weights at high utilization and add only enough compute to feed the stream. The most interesting US player is Positron, shipping the Atlas inference machine and building the Asimov chip with weights next to multipliers using commodity memory. Trade-off: not programmable like GPUs; exposed as an OpenAI-compatible API instead.
More from Infra
- Ex-OpenAI researcher Kokotajlo: robot-built factories could double compute every 3 months — victor_explore · 2026-09-11
- DeepSeek V4.1 Flash reportedly bakes prefill/decode disaggregation into the model weights — altryne · 2026-09-11
- Oracle shares rally as it avoids more debt; analyst sees it as commodity GPU business — TiernanRayTech · 2026-09-11
- 97.5% cache hit rate still hid half the bill: real numbers from six agents — Icy_Comfort_6220 · 2026-09-11
- Huawei Unveils Near-Package Optics Standard in Challenge to Nvidia and Broadcom — pstAsiatech · 2026-09-11
- Running Redis as a simple cache? Turn off snapshotting and journalling to save disk and CPU — DanielLockyer · 2026-09-11