a16z Essay: AI Inference Demand to Drive Shift from General GPUs to Custom Silicon
a16z Newsletter · rss · 2026-07-23
A deep dive essay from a16z argues that AI inference is rapidly overtaking model training as the largest workload in computing history, driving a historic shift from general-purpose GPUs to custom specialized hardware.
Key takeaways include:
- Inference as the Major Cost: While training is a one-time capital expenditure, inference is an operating expense that scales linearly with usage. Generating a single token requires massive memory reads and matrix arithmetic. With the explosion of users and agent platforms at OpenAI and Anthropic, the demand and cost for inference compute are going vertical.
- Bottlenecks of General-Purpose GPUs: GPUs launched the AI era due to their flexibility, but they are fundamentally mismatched for the highly specific workload of inference. To maximize throughput, the industry relies on "batching," which inherently increases latency—a tradeoff unacceptable for real-time coding agents. Furthermore, as data centers become power-constrained, watts wasted on a GPU's general-purpose flexibility directly cut into token-generating revenue.
- The Rise of Custom Silicon: Just as graphics processing migrated from CPUs to GPUs, massive and stable workloads inevitably earn their own specialized hardware. Google's TPU is an early example, but the trend is now accelerating across hyperscalers and frontier labs.
- Etched's Bet: The essay highlights the startup Etched, whose founders dropped out of Harvard before ChatGPT to build an end-to-end inference system—from chips to racks. Their core technologies include low-voltage inference (packing more compute into the same power envelope) and cluster-scale memory (pooling memory across an entire rack via ultra-low latency interconnects). Etched has hired over 400 engineers, operating on the philosophy that "production is the product."
More from Venture
- 8 open-source projects you can turn into income: n8n, Supabase, Ghost and more — Shruti_0810 · 2026-09-11
- A 160k-word knowledge base landed a $280k GEO contract — but the service model barely scales — sujingshen · 2026-09-11
- Indie hackers aren't just engineers or marketers — AI lets one builder run the whole loop — alexmacgregor__ · 2026-09-11
- Supabase grew ARR from $1M to $170M in 5 years, now valued at $10.5B — FinanceYF5 · 2026-09-11
- Glean Is Worth $7.2B, but What's Actually Its Moat? — yogthinks · 2026-09-11
- VC advice for Indian founders: stop pitching that you'll be in SF, go where customers are — vaibhavbetter · 2026-09-11