DeepSeek open-sources DeepInfer, the engine serving 155T tokens daily
Prompt Engineering · youtube · 2026-10-21
A Prompt Engineering video breaks down DeepInfer, the newly open-sourced inference engine behind DeepSeek's API, which serves 155 trillion tokens a day on a single cluster running DeepSeek V4.1 Flash. Key details:
- Tokenization and prompt-injection protection via deepseek-recipe
- KV cache retained for days: 96% of input tokens are cache hits
- Prefill and decode run on separate machines; expert parallelism across 64+ GPUs
- DSpark speculative decoding and batch-invariant inference
The author also hands-on tests deepseek-recipe and the temperature-0 batching effect on a Mac, and explains why DeepInfer won't run on a DGX Spark. DeepInfer and deepseek-recipe are on GitHub, V4.1 Flash 1017 weights are on Hugging Face, and DeepSeek's deployment write-up is available in Chinese and via Latent Space's English translation.
More from Infra
- GitHub hit by delays for Pull Requests and Webhook deliveries — DanielLockyer · 2026-10-21
- NVIDIA expands supply chain: substrate capacity to grow 2.5x, Amkor packaging up 300% — nvidia · 2026-10-21
- Node.js 24.22.1 Docker images finally published after hours-long delay — DanielLockyer · 2026-10-21
- Micron pitches co-design memory roadmap as key to meeting AI demand — BenBajarin · 2026-10-21
- Memphis votes to temporarily ban new data center construction for 135 days — Polymarket · 2026-10-21
- Micron details AI-class NAND: high-IOPS and high-bandwidth lines for KV cache and long context — BenBajarin · 2026-10-21