2,100+ tok/s/user for DeepSeek V4.1 Flash on 8× AMD MI355X, 2.8x faster than open-source
JiaZhihao · x · 2026-10-28
An engineering deep-dive on low-latency LLM inference on AMD MI355X: three architectural challenges create coordination overhead — no PDL-equivalent in HIP to overlap dependent kernels, distributed L2 caches adding inter-die traffic, and no NVSwitch-style in-network reductions causing up to 4x more inter-GPU communication. The team co-designed inference engine, kernels, and compiler to tackle this 'coordination tax', reaching 2,100+ tokens/s/user for DeepSeek V4.1 Flash at full quality on 8× MI355X — 2.8x faster than existing open-source solutions — with roofline analysis pointing to further headroom.
More from Infra
- Amazon, Google and Microsoft both buy from Anthropic and sell it $269B of compute — YvesMulkers · 2026-10-28
- AMD and Meta deepen partnership: Helios rackscale systems deployed for Meta's superintelligence infra — wkmyrhang · 2026-10-28
- DeepSeek on $75K of hardware: 2,120 tok/s decode at 16 concurrent sessions — victormustar · 2026-10-28
- Back-of-envelope math: 120B DeepSeek-class tokens per person by end of 2027 — bookwormengr · 2026-10-28
- Google TPUs to Run on Oracle Cloud, Making OCI the First Outside Hyperscaler With TPU — rseroter · 2026-10-28
- Arm EVP Mohamed Awad on why agents need their own CPU, a $251B server CPU market by 2030 — BenBajarin · 2026-10-28