Debate over local inference efficiency, centralization economics and MFU as a metric
A technical debate between cisfemale and mike64t over why local (on-device) inference is inefficient, and whether GPU resources should therefore be centralized.
Confirmed
- In an explainer post, cisfemale attributed local inference inefficiency to its root cause: running a model requires moving weights from memory to the processor, which costs about 100x more than the inference compute itself (FLOPs); with 100 users sharing, the weights are moved once and 100 computations are run, cutting costs roughly 50x — the core rationale for batching.
- mike64t accepted the physics of that cost accounting but pushed back on the "therefore centralize" conclusion: in the CPU world, almost no software runs anywhere near optimal.
- cisfemale countered with a comparison: CPU workloads are fragmented and software is complex, so little runs near optimal; in the GPU world, compute is expensive and the software stack simple, so things run quite close to optimal — MFU (Model FLOPs Utilization) above 30% is common.
- mike64t raised two counterpoints: commercially, consumer demand is the long-term mass revenue source (analogous to SpaceX earning more from Starlink than rocket launches), and as long as hardware depreciates, rent-seeking sharing is not a good strategy in the limit; technically, the achievable MFU ceiling varies with model size, precision, sharding approach, and other factors — MFU is simply not comparable across models, making it a "meaningless metric," and tokens per second per GPU (tok/s/GPU) is the only north-star metric worth optimizing.
Why it matters
- The debate cuts to the core economics of local inference vs. centralized cloud services: the 100x cost gap between memory movement and compute is the theoretical basis for batching and shared deployment, but the argument shows this doesn't necessarily imply "must centralize."
- The disagreement over metrics (MFU vs tok/s/GPU) has direct practical value for inference infrastructure teams choosing optimization targets: setting uniform MFU expectations can mislead, since the achievable ceiling depends heavily on model size and deployment details.
2026-10-30 ~ 2026-10-30 · 6 related posts
- Episode 1: Altman calls local inference a waste of DRAM, sparks backlash; reveals internal model cracking researchers' hardest problems(2026-10-29, 37 posts)
- Episode 2: Debate over local inference efficiency, centralization economics and MFU as a metric(2026-10-30, 6 posts)
Primary sources
- [source] Why local inference is inefficient: moving weights costs ~100x more than computing — cis_female · 2026-10-30
- Why Local Inference Is 'Inefficient' — and Why That Doesn't Matter — mike64_t · 2026-10-30
- GPU-World Runs Near Optimal: MFU Above 30% Is Common, Argues cis_female — cis_female · 2026-10-30
- [source] MFU Is a Meaningless Metric — tok/s/gpu Is the Only One That Matters — mike64_t · 2026-10-30
- [source] Infra practitioner: MFU is a meaningless cross-model metric, tok/s/GPU should be the north star — mike64_t · 2026-10-30
- GPU vs CPU economics: MFU above 30% is common, but is it FLOPs well spent? — joseph_h_garvin · 2026-10-30