Running DeepSeek V4 Flash 155G on DGX Spark: 2-bit Quantization & MTP Benchmarks
Puzzleheaded_Base302 · reddit · 2026-08-03
The author details the process and benchmarks of running the 155 GB DeepSeek-V4-Flash-0731 locally on a single DGX Spark (GB10, 121.7 GiB unified memory) using vLLM-Moet with 2-bit quantization.
Key Performance Metrics:
- Steady prefill rate at 1000 t/s.
- Enabling MTP (Multi-Token Prediction) boosts decode speed from 19 tok/s to 25.2 tok/s (+31.5% at concurrency 1), though aggregate throughput sees a slight 5.3% drop at high concurrency (conc 4).
Critical Deployment Gotchas:
- Native Build Required: Docker images cannot be transferred across architectures (x86 to ARM64); they must be built natively on the Spark.
- Dockerfile Fix: The default base image digest is amd64-only; it must be swapped to a multi-arch tag.
- Hardware Adaptation: Code must be modified to bypass unsupported sm121 GPU errors, and a massive swapfile is needed to prevent loading failures.
- Timeout Handling: vLLM's default timeout limits must be increased due to extensive loading times.
Related event: DeepSeek V4 Flash Sparks a Wave of Local Deployment Tests(26 posts)→
More from Infra
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16