DeepSeek V4.1 Flash runs at 20 tok/s on a single RTX 3090 with 64GB RAM
casper_hansen_ · x · 2026-11-12
casperhansen reports DeepSeek V4.1 Flash 1017 running at 20 tokens/s on a single RTX 3090 with 64GB RAM, with no quality loss. Stack: vLLM frontend for scheduling, tool calling and OpenAI/Anthropic-compatible APIs, ExLlamaV3 for the forward pass. Shared DSpark speculative decoding benchmarks show 21.3 tok/s decode at 8K prompt in nvme mode, and time-to-first-token after an 8K cached prefix beating GLM-5.3-Flash (8.0s vs 8.9s for 400 new tokens). He plans to replace ExLlamaV3 with his own forward pass.
More from Infra
- Gemini Decision 1 costs $345 per 1M tasks, 3x pricier than leader Drex Max v1.1 — ArtificialAnlys · 2026-11-12
- Cloudflare's weekend-trained hybrid model cuts agent cost from 40K to 1.8K tokens — TheTuringPost · 2026-11-12
- Microsoft paper details Maia accelerator and software-defined dataflow for trillion-token-scale serving — thoefler · 2026-11-12
- LMSYS megakernel pushes decoding to 1,109 tok/s on 8× GB300, launches cut 1450→97 — ying11231 · 2026-11-12
- Dev builds all-rust AI OS LexiSuite, claims faster Qwen3 27B inference than Ubuntu — W61k3r · 2026-11-12
- OpenRouter's new routing algorithm cuts GLM 5.3 Flash TTFT 40% and price 17% — AccBalanced · 2026-11-12