Running DeepSeek V4 Flash 155G on DGX Spark: 2-bit Quantization & MTP Benchmarks
Puzzleheaded_Base302 · reddit · 2026-08-03
The author details the process and benchmarks of running the 155 GB DeepSeek-V4-Flash-0731 locally on a single DGX Spark (GB10, 121.7 GiB unified memory) using vLLM-Moet with 2-bit quantization.
Key Performance Metrics:
- Steady prefill rate at 1000 t/s.
- Enabling MTP (Multi-Token Prediction) boosts decode speed from 19 tok/s to 25.2 tok/s (+31.5% at concurrency 1), though aggregate throughput sees a slight 5.3% drop at high concurrency (conc 4).
Critical Deployment Gotchas:
- Native Build Required: Docker images cannot be transferred across architectures (x86 to ARM64); they must be built natively on the Spark.
- Dockerfile Fix: The default base image digest is amd64-only; it must be swapped to a multi-arch tag.
- Hardware Adaptation: Code must be modified to bypass unsupported sm121 GPU errors, and a massive swapfile is needed to prevent loading failures.
- Timeout Handling: vLLM's default timeout limits must be increased due to extensive loading times.
More from Infra
- 30B Video MoE Quantizations Tested: Most Users Should Wait — EntireBig7258 · 2026-08-03
- AI Data Centers Consume Up to 1.5 Billion Gallons of Water Yearly — AndyMasley · 2026-08-03
- Prepping for Local LLM Inference: Enthusiast Builds 30TB SSD & 256GB RAM Rig — reto-wyss · 2026-08-03
- CPO Packaging Tech Unlikely to See High-Volume Shipments Before 2028 — BenBajarin · 2026-08-03
- DeepSeek V4 Flash Prefill Speed Boost: Downgrade to CUDA 13.1 — fragment_me · 2026-08-03
- Cornell Releases Roadmap for Parallel Programming and HPC Concepts — thehiphopswami · 2026-08-03