Why AI chips move trillions of bits for cheap arithmetic
prateekj · x · 2026-08-24
AI chips face a bottleneck because conventional computers physically separate memory and compute, requiring massive data movement (e.g., 100GB per token for a 100B model) for relatively cheap arithmetic. Potential solutions include:
- Compute-in-memory: Placing simple arithmetic inside or next to memory.
- Move less data: Compressing weights, using lower precision, or avoiding irrelevant parts of the model.
- Reuse data: Caching frequently-used values in fast local memory.
- Sparse models: Activating only a small subset of parameters needed for each token.
- New architectures: Moving beyond Transformers to architectures that retrieve small, relevant knowledge pieces instead of running through enormous matrices.
More from Infra
- DeepSeek Local Deployment: 4x W7900 vs 2x Blackwell for Agentic Workflows — Retumbo77 · 2026-08-24
- Running Local LLMs on Shared Machines with Zero Data Retention — Developer-Y · 2026-08-24
- 3D DRAM touted as more efficient and higher bandwidth than HBM4 — zephyr_z9 · 2026-08-24
- UK data center water use is a fraction of leak losses — Yamapama · 2026-08-24
- PSA: Commercial GPUs offer marginal speed gains for video generation despite high cost — mwoody450 · 2026-08-24
- AI Sandbox Escape: C++ Bridge Bug Bypasses V8 Isolation — VoidStateKate · 2026-08-24