DeepSeek v4.1 Flash served at 103 tok/s on 2x DGX Spark with vLLM
EAccelerate_42 · x · 2026-10-12
@sudoingX served DeepSeek v4.1 Flash on two NVIDIA DGX Spark boxes with vLLM, setting a speed record:
- 141.3 tok/s structured output, 103.0 tok/s code, 58.3 tok/s prose — 2.5x the best prior vLLM setup, beating tensorfold v0.6.0 on code/structured output
- 262k-token context per request; 3/3 needle-in-haystack hits at 8k–256k, decoding 76 tok/s even after a 255,709-token prompt
- Prefill 1,395–1,475 tok/s (admitted weak spot); experts quantized to 2.70 bits vs 2.9 bits for the rival build, self-quantized from the official checkpoint and scored against native DeepSeek
A concrete recipe for running huge MoE models on workstation hardware via aggressive quantization plus vLLM tuning.
Related event: 2x DGX Spark Runs DeepSeek v4.1 Flash at Record 103 tok/s for Code(2 posts)→
More from Infra
- OpenAI chip head Richard Ho on Jalapeno: why Broadcom beat Nvidia and AMD as partner — richardjnieva · 2026-10-12
- Bittensor's TPN Labs ships first model TPN005, cutting base size ~40% with 32k context intact — const_reborn · 2026-10-12
- Microsoft to scale in-house Maia AI chips to multi-gigawatt inference by 2027 — The Decoder · 2026-10-12
- Microsoft plans huge deployment of its own AI chips to cut costs — pstAsiatech · 2026-10-12
- Ragged KV cuts Looped LM inference FLOPs by 25% with no retraining — DanielKhashabi · 2026-10-12
- NVIDIA's ELR removes the one-loop tax: Koi-4B scores 18 at one loop, 29 at four loops — nvidia · 2026-10-12