Streaming inference stack runs full-precision DeepSeek V4.1 on one 5090 at 1770 tok/s prefill
Front_Eagle739 · reddit · 2026-10-28
A developer open-sourced streaming-llama-stack, a streaming/disaggregated inference stack for small-memory systems with fast SSDs. On a single RTX 5090 + 32GB DDR5 + 25GB/s RAID0 it runs full-precision DeepSeek V4.1 at up to 1770 tok/s prefill and 7-20 tok/s decode.
Key ideas
- DirectStorage streams oversized models from NVMe for prefill; decode uses router biasing to keep hot experts in VRAM for 20 tok/s, or streams most experts at 7 tok/s full precision.
- Disaggregated prefill/decode: prefill on the RTX box, pass KV/state over the network to a big-memory Mac or server, with automatic crossover selection by prompt length.
- WebUI offers a dynamic router-bias slider (max quality ↔ max speed) and per-phase settings (fast reasoning, max-quality code blocks).
- Full-precision MiniMax H3 and Magi 2 Preview ran off one 5090 at near-resident speed; includes a 20-minute single-GPU identity LoRA recipe and cross-engine mixing (streaming prefill + MLX decode with KV translation).
Limits: streaming decode currently only DSV4.1 (GLM 5.3 Flash in progress); streaming prefill supports GLM 5, Step 3, Kimi K2.5/6/7; Windows-only, Blackwell-class GPUs only.
More from Infra
- Mixed NVIDIA+AMD multi-GPU LLM setup benchmarks slower than a single card — LMasterGame · 2026-10-28
- 8 open-source repos to self-host a lean AI agent stack instead of stacking $20/month SaaS — SaiTarunRonanki · 2026-10-28
- Corning Optical Q3: sales up 44% to $2.38B, Gen AI revenue doubles y/y — demian_ai · 2026-10-28
- AI-powered formal verification finds real Redis/Valkey bugs, incl. a 23-char freeze — LingmingZhang · 2026-10-28
- 112 TPS on a single RTX 3090: LlamAmpere v0.6 pushes 27B models with 400K context — Brief-Tap-6616 · 2026-10-28
- Your coding agent burns up to 12k tokens before you type: 30 harnesses span 50x — yb2698 · 2026-10-28