Betelgeuse Runs DeepSeek V4.1 Flash Q8 at 62 tok/s Prefill on a Single RTX 3060 12GB
zyxciss · reddit · 2026-11-08
A developer's Betelgeuse engine runs DeepSeek V4.1 Flash Q8XL (305.6GB experts + 110.6GB Engram table) on an RTX 3060 12GB with only 16GB RAM, hitting 19 tok/s decode and 62 tok/s prefill.
The bottleneck and fix
- Out-of-core MoE inference suffers brutal TTFT: a prompt activates nearly every expert across 40 layers, so naive prefill crawls at 1-6 tok/s.
- Three changes: (1) location-invariant cache partitioning with 20 resident slots per layer (5 VRAM hot + 15 host-RAM); (2) single-pass layer-major prefill using 16-token ubatches, streaming host-RAM experts over PCIe in parallel with compute; (3) physical VRAM consolidation of the top-5 most-active experts after the first pass.
- The 110GB Engram table lives entirely on NVMe—it touches only 48 rows (192KB) per token, staged into a 2MB VRAM buffer.
Results: warm prompt processing jumped from 3-6 to 61.80 tok/s, decode peaks at 27.92 tok/s, agent-work decode 24.22 tok/s, peak memory 10.61GB, 0% cold SSD reads. Dual-channel RAM would double PP again. Beats Strata's decode speed on the same GPU with 4x the RAM.
More from Infra
- Single RTX 5090 serves 125B MoE at 256k context: QuixiInfer hits 27 tok/s decode — QuixiAI · 2026-11-08
- DeepSeek v4.1 Flash runs locally at 27 tok/s decode with 256k context on an RTX 5090 — QuixiAI · 2026-11-08
- Vercel lists DeepSeek V4.1 Flash 1017 at 1,000+ tok/s, but provider speed numbers don't add up — gajesh · 2026-11-08
- RL staleness in agentic training: >100-step off-policy regimes favor labs with huge compute — teortaxesTex · 2026-11-08
- Inside China's 'Stargate': an 800MW Inner Mongolia campus built at one-fifth of US cost — dylan522p · 2026-11-08
- Thieves steal $11.4M of Nvidia hardware with a fake food-delivery truck — eyishazyer · 2026-11-08