GPU dominance in deep learning is largely accidental: the case for custom chiplet inference hardware

ai · x · 2026-09-30

A 43-minute deep dive on fleetwood.dev argues that GPUs' dominance in deep learning is largely accidental—a serendipitous overlap between graphics and ML workloads—and that their graphical heritage still shapes today's architectures. Citing Onur Mutlu's observation that over 90% of system energy in large ML models goes to memory, the author contends that "compute-centric" thinking is a bad model for AI inference.

Key points:

With Moore's Law and Dennard scaling slowing, energy and capital constraints leave hardware architects no choice but domain-specific architectures (DSAs) designed backwards from the Transformer workload
Assumptions like power-of-2 matrix sizes and dmodel dimensions are artifacts of CUDA warps and HBM limits, not information-theoretic optima
Training/inference silicon disaggregation is just the start; massive model disaggregation via chiplets with advanced packaging and interconnects is coming
The piece predicts how the inference paradigm will shift, framed as essential foresight for hardware architects and investors

The reposter quips that current LLM design amounts to "hardware Stockholm syndrome," with custom chiplets and interconnects as the next frontier.

Original post →

GPU dominance in deep learning is largely accidental: the case for custom chiplet inference hardware

More from Infra