DeepSeek V4.1 on a single RTX 5090: 4000+ tok/s prefill, 7-24 tok/s decode via NVMe streaming
Front_Eagle739 · reddit · 2026-10-30
A Reddit user open-sourced streaming-llama-stack, running fp4/fp8 quantized DeepSeek V4.1 locally on a single RTX 5090 with 32GB RAM and a dual PCIe5 NVMe RAID0, doubling their previous performance:
- 4000+ tok/s prefill on long prompts by streaming from the RAID array, then dropping prefill buffers and caching a hot set of experts
- 7.5 tok/s decode with no divergence from other engines; missed NVMe-resident experts fetched via heavily optimized DirectStorage
- A per-token speed/quality slider trading speed vs KL divergence: 1.5bpw-equivalent KL yields 24 tok/s; profiles for fast reasoning, quality code/tool calls, and balanced answers
- KV can be offloaded to a Mac for decode; the same streaming system serves image and video generation in one chat
Faster NVMe helps proportionally; only tested on the author's machine so far.
More from Infra
- Intelligence per dollar fell ~1,000,000x in three years, shows double-exponential chart — banteg · 2026-10-31
- AWS grows 39.3% to $45.98B as analyst says AI capex panic misses the real cloud story — DavidLinthicum · 2026-10-31
- Joke startup idea: cannonball-powered turbines running a data center at the bottom of a pool — djcows · 2026-10-31
- Shipping a $380M High-NA EUV machine: 40 containers, 3-4 Boeing 747s — Yamapama · 2026-10-31
- China could flood InP laser market next year, single vendor capacity cited at multiples of demand — jwt0625 · 2026-10-31
- Why SanDisk outvalues Kioxia: a JV where one side owns all the fab downside — PAstynome · 2026-10-31