Infra practitioner: MFU is a meaningless cross-model metric, tok/s/GPU should be the north star
mike64_t · x · 2026-10-30
An infra practitioner argues MFU shouldn't be chased as a north-star metric: practical upper limits differ across model size, precision, and sharding, making MFU non-comparable and effectively meaningless—tok/s per GPU points in the right direction instead. He adds a business angle: end consumers drive large-scale revenue long-term (SpaceX earns more from Starlink than launches), and rent-seeking is a losing play while hardware keeps depreciating.
More from Infra
- You can fingerprint your exact GPU with a single torch.index_add_ call — CatAstro_Piyush · 2026-10-31
- SGLang boosts DeepSeek V4.1 Flash throughput 67% to 1.09M tok/s per GPU in 4 days — BanghuaZ · 2026-10-31
- llama.cpp ROCm regression: prompt processing drops from ~1200 to ~460 t/s on RX 9070XT — uber-linny · 2026-10-31
- Sub-scale neocloud warning: VC-backed GPU cloud startups reportedly struggling — AccBalanced · 2026-10-31
- Qwen4 with HC architecture could widen the Q4 vs Q6 quantization gap significantly — Ok-Shower7286 · 2026-10-31
- Too many inference providers, too little demand: a developer's take — AccBalanced · 2026-10-31