MIT Tech Review: data movement is the new AI infrastructure bottleneck

MIT Tech Review AI · rss · 2026-09-05

MIT Technology Review, with Tirias Research analyst Jim McGregor, argues AI inference is reshaping data center architecture:

Inference changes the optimization problem: AI is millions of distinct workloads, not one — memory bandwidth, storage throughput, and networking can't be optimized in silos; the data center must be designed as an integrated system.
Data movement is the new bottleneck: RAG-style workloads demand constant retrieval and caching across massive databases, elevating memory and storage from background hardware to strategic assets. Bottlenecks migrate between layers, so compute, memory, storage, and networking must be architected together.
Latency is business value: in healthcare, finance, and customer-facing AI, delays undermine safety and trust, making infrastructure performance a reputational issue.
Procurement framework: define actual workloads (not generic "AI readiness"), build modular architecture, work with the full supplier ecosystem, reassess continuously, and optimize for efficiency and ROI over peak performance.

Bottom line: the winners won't be those with the largest clusters, but those best able to align every infrastructure element to their AI workloads.

More from Infra