DeepInfer lacks KV caching, load balancing and disagg, but 3-4K GB200s can match DeepSeek's reported throughput
teortaxesTex · x · 2026-10-21
teortaxesTex weighs in on DeepSeek's open-sourced DeepInfer inference framework, endorsing a critique by @Halex623 as one of the few objective ones. The release is missing three critical pieces for at-scale serving:
- No KV caching — essentially a non-starter for inference; 3FS offload exists but requires setting up a distributed file system.
- No load balancing — you must bring your own routing stack (e.g. NVIDIA Dynamo or SMG).
- No disagg machinery — transfer logic is unimplemented, tricky to get right across clusters with heterogeneous networking.
The author notes DeepSeek does ship 3FS and doesn't optimize for heterogeneous setups, but concludes that with 3-4K GB200s you can roughly match the officially reported server throughput.
More from Infra
- vLLM's specialized decode path hits 829 P90 TPS per user on four B300 GPUs, 26% faster — vllm_project · 2026-10-21
- vLLM says it can reproduce DeepSeek's inference performance, yielding $39/GPU/hr revenue — vllm_project · 2026-10-21
- vLLM matches DeepSeek's DeepInfer decode performance within 6%, generating $39/GPU/hr at off-peak pricing — vllm_project · 2026-10-21
- AMD brings EAGLE-4 speculative decoding to Instinct MI355X day 0, 1.6x faster than built-in MTP — hongyangzh · 2026-10-21
- TokenGate: learned token pruning speeds up LLM prefill up to 11.3x — Liana Patel · 2026-10-21
- Modal revives multiplayer game Curvytron to demo its new Sticky Sessions routing — charles_irl · 2026-10-21