Indie dev builds DeepSeek V4.1 Flash inference engine with 512K context and on-demand KV cache
gnukeith · x · 2026-11-08
- Developer gnukeith shares details of his self-built inference setup: 3GB hard-reserved for the OS to prevent crashes, DeepSeek V4.1 Flash at 512K context by default (extendable to 1M, honored by the server), and KV cache allocated on demand rather than upfront.
- He caps it at 7 concurrent instances (could push to 10) to avoid overloading. The design borrows mainly from DeepInfer and vLLM, plus engines like Gufo and Halogen-Flash.
- He's considering open-sourcing the engine around late November/early December, though it's heavily tailored to DeepSeek V4.1 Flash and his niche setup.
More from Infra
- François Fleuret asks: after a loss spike and recovery, is final loss worse, same, or better? — francoisfleuret · 2026-11-08
- ResearchBoost Agent speeds up real research workloads 2.51x on average across 12 tasks — YouJiacheng · 2026-11-08
- Dev builds Cray-2 emulator from scratch, runs a GPT-3-class LLM at 1 token per 4s — inductionheads · 2026-11-08
- Hyperbolic CEO on compute market-making: bridging 3-year contracts and 3-week needs — YiMaTweets · 2026-11-08
- Ben Bajarin mocks 'we don't need data centers' take: CSPs were 'useless' too — BenBajarin · 2026-11-08
- KV Cache Quantization Tested Across 35 Quants: 8-bit Is Essentially Free for Qwen3.8 27B — Hefty_Wolverine_553 · 2026-11-08