DeepSeek V4.1 Flash at 2.61 bpw hits 26 tok/s on dual RTX 3090s with dsx
whiteh4cker · reddit · 2026-10-19
A Redditor used lxe's dsx engine (Rust + CUDA, built for 2-bit quantized MoEs on 3090s) to create a 2.6127 bpw quant of DeepSeek-V4.1-Flash, benchmarked on 2×RTX 3090 + 192GB DDR5:
- Quality: vs. the author's 2.03 bpw build, KL divergence drops from 0.462 to 0.1972, Top-1 retention is 82.59%, PPL 9.36
- Speed: 26 t/s generation and 170 t/s prompt processing at 262K context with 203GB Engram tables on a SATA SSD
- Pain points: loading 187.6GB of weights from an HDD takes 90–110 minutes; dsx doesn't support images yet
- Hardware: 2×RTX 3090, Intel Ultra 7 270k, 192GB DDR5-5600
Useful reference for anyone running large MoEs locally on consumer dual-GPU setups for coding and agentic tasks.
More from Infra
- d-Matrix CEO on why 3D-stacked memory compute could break the AI memory wall — firstadopter · 2026-10-19
- Rubick: an Open-Source Gateway Serving Chat, Embeddings and Calibrated Decisions from One Model — wFXx · 2026-10-19
- Mixing a 9070 XT with an RTX 3080 for local LLMs: 30-50 tok/s and hard lessons — BrightCandle · 2026-10-19
- LLM Quantization Explained: Bit Width Alone Doesn't Determine Accuracy or Speed — techNmak · 2026-10-19
- 303 companies mapped on Nebius: from 1X robots to Mastercard, tracked in independent atlas — pdamodaran · 2026-10-19
- vLLM details MiniMax M3.1 Flash serving stack: 1.7x faster indexer decode via NVFP4 — vllm_project · 2026-10-19