LLM Inference Is Bandwidth-Bound: Lemire on Why Positron Inverts GPU Design
lemire · x · 2026-09-11
Daniel Lemire argues inference — not training — is the part that must get cheap, and most hardware wasn't designed for it. LLM inference is bandwidth-bound: massive matrix-vector multiplies against barely-reused weights, closer to video streaming than simulation. GPUs were built compute-first with HBM bolted on. The fix: invert the design — stream weights at high utilization and add only enough compute to feed the stream. The most interesting US player is Positron, shipping the Atlas inference machine and building the Asimov chip with weights next to multipliers using commodity memory. Trade-off: not programmable like GPUs; exposed as an OpenAI-compatible API instead.
More from Infra
- SemiAnalysis: OpenAI reportedly negotiating ~1GW TPU commitment while seeking sites — dylan522p · 2026-10-28
- I tested 11 video upscaling methods for AI video: local, Topaz and cloud compared — lopas99 · 2026-10-28
- Thermo-backed CTO says AI is in its dial-up phase, touts disaggregated compute for an ultrafast 'broadband moment' — beffjezos · 2026-10-28
- DeepSeek-V4.1-Flash Q2 runs at 41 tok/s on one RTX 3090, 90% token match vs API — musicurgy · 2026-10-28
- USearch v3 RC ships after 1,000 days, its libraries now embedded in 550K-star projects — viglovikov · 2026-10-28
- SK Hynix says memory demand will outpace supply through 2030 — Chemical_Hawk_6307 · 2026-10-28