DeepSeek V4.1-Flash Runs 502GB Model on a Single RTX 5090 at 5-21 tok/s
AccBalanced · x · 2026-09-12
A developer demonstrated DeepSeek-V4.1-Flash generating tokens on one RTX 5090 with 125.7 GiB RAM, while most of a 502GB GGUF lives on NVMe. Hot experts stream through VRAM/RAM, cold experts stay on SSD, and a 196B Engram memory is disk-backed. Key numbers: 552B backbone, only 8B active per input token and 16B per output token, 5.12 tok/s on new content (up to 21.27 tok/s with resident data), and 0.9967 logit correlation to the reference model. Commentators see this — like Qwen3.8-Flash-Next — as evidence model capacity is decoupling from active compute, a blueprint for running far larger models locally.
More from Infra
- Clover-35B-A3B Quantized to 29.4 GB Runs at ~42 tok/s on a 64 GB Mac — viveky259 · 2026-10-31
- Liquid Agent brings fully local LFM3 agent to Linux with one-command install — JosephJacks_ · 2026-10-31
- A fully local office AI stack: MiniCPM, Qwen3, LFM2.5 and Whisper — HowDevelop · 2026-10-31
- a16z: GPU rental prices up 106% in 3 months while token prices fall 58% — a16z · 2026-10-31
- vLLM nightly adds PLE disk offload, with a telling [Qwen4Exp] tag on the PR — rmhubbert · 2026-10-31
- Solo AI studio: per-minute GPU rentals rendered a 328-frame Blender shot for ~$1 — r0ck3t23 · 2026-10-31