DeepSeek launches DeepInfer with open prefill/decode benchmarks and reproducible scripts
Xianbao_QIAN · x · 2026-10-20
DeepSeek released DeepInfer, its new inference serving stack, along with performance benchmarks: on 4 SM100 GPUs within one scale-up domain, prefill reaches up to 65,536 non-cached input tokens/s/GPU; decode on 64 GPUs delivers 22,800 output tokens/s/GPU at 112 tokens/s per user, tradeable toward 320 tokens/s/user interactivity. Config uses EP64DP64 with 384+64 experts and 131k context, with runsweep.sh provided for reproduction. The goal: make AI tokens accessible and cheap everywhere.
More from Infra
- NVIDIA's $500B AI infra deal had "no obvious provenance" as PE debt talks stall — mjdramstead · 2026-10-20
- The Local Computing Rebellion: Subscription Fatigue and Forced AI Drive a Shift Back On-Premises — DavidLinthicum · 2026-10-20
- DeepSeek also open-sources deepseek-recipe, replicating official API token encoding logic — deepseek_ai · 2026-10-20
- DeepSeek open-sources DeepInfer, the production inference framework behind DeepSeek-V4.1-Flash — deepseek_ai · 2026-10-20
- Scott Hanselman serves local inference to his whole house from a Surface Laptop Ultra — lee_stott · 2026-10-20
- DeepSeek-V4.1-Flash quantized to 2-4 bit mixed precision for dual RTX PRO 6000 with 0.1306 KL divergence — TheZachMueller · 2026-10-20