DeepSeek V4.1 Flash runs at 20 tokens/s on a single RTX 3090 with 2.25bpw EXL3 quantization
casper_hansen_ · x · 2026-11-12
casperhansen demonstrates DeepSeek V4.1 Flash 1017 running at 20 tokens/s on a single RTX 3090 with 64GB RAM, using a vLLM frontend (scheduling, tool calling, OpenAI/Anthropic APIs) with ExLlamaV3 for the forward pass on a 2.25bpw EXL3 quantized model. He clarifies "zero quality sacrifice" means no tokenmax-style quality tradeoffs, and plans to write his own forward pass eventually.
Related event: DeepSeek V4.1 Flash Runs at 20 tok/s on a Single RTX 3090(3 posts)→
More from Infra
- WSJ reports NVIDIA board preparing to oust Jensen Huang as CEO — NinaDSchick · 2026-11-12
- LeapQuant: training-free quantization makes linear attention up to 4.25x faster — tianyin_xu · 2026-11-12
- Microsoft's MAI Transcribe 2 sees 12x usage growth, hitting 8.5M weekly transcriptions — mustafasuleyman · 2026-11-12
- gVisor joins CNCF with Modal as co-maintainer after upstreaming GPU and training support — josh_wills · 2026-11-12
- Local vs Cloud AI Coding Agents: Benchmarks, Hardware Costs, Performance — luisdans · 2026-11-12
- Agent Substrate 1.0 GA: microVM-sandboxed agent runtime with zero ambient credentials — rakyll · 2026-11-12