AMD MI355X beats NVIDIA B300 by 24.4% throughput serving Kimi K3, 87% cheaper per token
vllm_project · x · 2026-10-17
The vLLM team published a deep dive on serving Kimi K3 on AMD Instinct MI355X. On SemiAnalysis' AgentX agentic coding benchmark at 50 tok/s/user, 8× MI355X delivers 7,567 vs 6,083 tok/s per GPU on HGX B300 — 24.4% more throughput — and 87.4% more tokens per dollar under the benchmark's ownership-cost model.
Key optimizations covered:
- Cache: keep conversation state available for reuse
- MoE: remove duplicate expert work across GPUs
- Memory traffic: fuse ops, eliminate unnecessary copies
- MLA: faster long-context decoding with AMD AITER assembly
- KDA: parallelize prefill and speculative verification
- Speculative decoding: tune how many tokens to draft
The post also teaches patterns for spotting bottlenecks in your own inference stack. AMD matching NVIDIA per-GPU throughput is a notable signal for compute economics.
Related event: AMD MI355X beats NVIDIA B300 by 24% in Kimi K3 inference throughput(2 posts)→
More from Infra
- Texas grid queue hits 474GW; white paper argues AI data centers should become grid resources — espricewright · 2026-10-17
- HF Transformers adds activation checkpoint offloading to CPU to free GPU memory — StasBekman · 2026-10-17
- DatologyAI claims 1000x compute-efficiency headroom from data curation, demo on Oct 27 — josh_wills · 2026-10-17
- Supabase ships MIT-licensed Middleware: write fetch logic once, run on any runtime — dshukertjr · 2026-10-17
- Dev asks for sandbox provider supporting N concurrent sandboxes with dynamically mountable disks — zeeg · 2026-10-17
- Litespark serves Nemotron 120B at 962 tokens/sec per user on just two H200 GPUs — NVIDIAAI · 2026-10-17