DeepInfer repo needs 64 SM100 GPUs for decode, still lacks a full serving stack
AIFlow_ML · x · 2026-10-21
AIFlow breaks down what the DeepInfer repo does and doesn't include. In the repo: kernels and scheduler, DSpark speculative decoding, SWA bounded replay, batch-invariant inference, and a deepseek-recipe for API parity. Missing per the README: KV cache system, prefill/decode disaggregation, load balancing across replicas, an OpenAI-compatible server, and multi-model support. Benchmarks ran on 4 SM100 GPUs for prefill and 64 for decode, with Huawei Ascend 950 supported — but no independent numbers yet. The ideas are expected to land in vLLM and SGLang first. The standout feature for self-hosted stacks: batch-invariant inference, where your output doesn't change based on other requests in the batch — a big deal for debugging agents.
More from Infra
- Data center power set to double by 2030 as grid queues top 2,500 GW — ingliguori · 2026-10-21
- antirez tries DHH's Omarchy on a Dell XPS, plans to adapt DwarfStar SSD model streaming — antirez · 2026-10-21
- NVIDIA reportedly sits on $35B in China orders but can supply only 5-10% — emmanuelvivier · 2026-10-21
- Nvidia's China orders reportedly hit $35B, but supply may cover just 5–10% — emmanuelvivier · 2026-10-21
- willcb: open-sourcing a frontier lab's prod stack is cool, but nothing majorly new today — willcb · 2026-10-21
- swyx: AI compute is supply-constrained — more tokens means more revenue, not lower prices — swyx · 2026-10-21