DeepInfer is deeply tailored to DeepSeek 4.1F: KV cache of just 890 bytes per token
teortaxesTex · x · 2026-10-25
- An insider explains that DeepSeek's open-source inference framework DeepInfer is deeply customized for the 4.1F model: task assignment ignores VRAM capacity because 4.1F's KV cache is extremely optimized — 890 bytes per token, so a full 1M-token context fits in under 1GB (previous 4F needed 4x).
- This tiny KV cache lets DeepInfer throttle purely by batch size. By contrast, Kimi K3.1 needs 27kB per token in bf16 (30x overhead, 15x in FP8), plus a larger model and activations — making long-context serving very hard.
- A quoted comment argues Kimi's weak inference infra burns through subscription quotas fast, hurting an otherwise strong model.
More from Companies & People
- Pichai: RSI is key to Google's Argon model, but we're far from its end state — haider1 · 2026-10-25
- Musk digs up 10 failed Starlink doomer predictions from 2021-2023 via Grok — elonmusk · 2026-10-25
- Remote engineer earns US startup salary from hometown, paying zero rent vs ₹45,000 in Bangalore — _jaydeepkarale · 2026-10-25
- Rambus signs multi-year deal to deploy ChipAgents across chip design workflows — WilliamWangNLP · 2026-10-25
- DeepSeek insider: infra is the big brother and can veto research decisions — teortaxesTex · 2026-10-25
- Claude pegs Chinese API distillation spending at $50–100M a year, ~0.01% of US GDP growth — JasonBotterill · 2026-10-25