Agentic Loops Shift AI Infra Focus From FLOPs to the von Neumann Bottleneck
sarahookr · x · 2026-09-03
Sarah Hooker argues that babysitting multi-step agentic loops requires constantly offloading intermediate states, so the future of AI infrastructure is not just maximizing FLOPs but innovating on the von Neumann bottleneck—how fast data can move back and forth. Implications:
- Datacenter designs will change.
- Hardware in space is probably not a good idea.
- It partly explains why consumer laptops have gone up in price—memory markets are extremely expensive right now.
More from Infra
- Auction-based LLM routing paper finds fixed token pricing wastes 10%-71% in costs — _onionesque · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- SGLang Summit sponsors jump from 11 to 36 in one week — ying11231 · 2026-10-18
- Oracle CDS hits record 300bps, called first AI trade domino after Barclays sell downgrade — arthurcolle · 2026-10-18
- Agent Coding Workflow Would Cost ~$40k/Month on API Pricing, Dev Reveals — ChrisRackauckas · 2026-10-18
- PS5LM: llama.cpp port turns a jailbroken PS5 into a home LLM server (500+ GitHub stars) — wateriscoding · 2026-10-18