Lithos engine hits 2,116 tokens/s per user on 8× AMD MI355X, up to 3.4× faster than OSS engines
JiaZhihao · x · 2026-10-27
Lithos AI announced what it calls the fastest inference engine on AMD GPUs. On eight AMD Instinct MI355X GPUs, it runs DeepSeek V4.1 Flash at 2,116 tokens/s per user (up to 3.4× faster than major open-source engines) and Kimi K3 at 652 tokens/s (up to 2.5× faster), while preserving full model quality and official precision formats.
Key technical points:
- The core problem is the "coordination tax" — time lost waiting between kernels and across GPUs
- Approach combines persistent megakernels, overlapping execution, and communication inside computation
- Leverages MI355X register capacity and direct instruction-level control for hardware-specific optimizations
- Deep dive covers decode speed, cost per token, latency up to 1M context, and roofline analysis
Benchmarks credit all engines the same 75% speculative acceptance rate; some competing configs use reduced precision or more GPUs.
Related event: AMD and Lithos Hit 2,100+ Tokens/sec per User with DeepSeek on MI355X(3 posts)→
More from Infra
- Arm EVP Mohamed Awad on why agents need their own CPU, a $251B server CPU market by 2030 — BenBajarin · 2026-10-28
- Jay Kreps: the frontier AI race has a built-in self-stabilizing dynamic — weak models rebuild compute — olcan · 2026-10-28
- LiquidAI Ships 2B Omni Open Model With Day-One llama.cpp Support — ilintar · 2026-10-28
- Mojo/MAX Hits Phones: 4-Day Project Matches llama.cpp Vulkan Inference — Cane_P · 2026-10-28
- EmbeddingGemma 2 runs real-time embeddings on a MacBook CPU alone: 20ms text, 100ms images — generativist · 2026-10-28
- Statistically lossless Qwen3.8-27B GGUF quantizations land on Hugging Face, analysis paywalled — Dany0 · 2026-10-28