DeepSeek-V4.1-Flash on 2x DGX Spark: TensorFold decodes 92 tok/s code, 2.1-2.3x faster than vLLM kit
shayanx45 · reddit · 2026-10-18
Running DeepSeek-V4.1-Flash-1017 (EXL3 2.9bpw, 1M context) on two NVIDIA DGX Spark units (TP=2 over CX7), the author benchmarks TensorFold against the vLLM kit on identical weights and prompts: 2.1-2.3x faster decoding—56.8 vs 27.0 tok/s text, 92.2 vs 39.9 tok/s code, 64K needle-in-haystack in 35.9s vs 80.4s, plus 4 concurrent requests at 140 tok/s total where the kit runs one at a time.
On top of bertholomus's fast TensorFold port, the author added agent usability: forced and named tool calls on DeepSeek's DSML format (previously 400 errors), prompts byte-identical to DeepSeek's reference encoder, jsonschema/regex/choice output on the speculative two-node decoder, and fault isolation so one bad request doesn't kill the lane.
Extras and caveats: prompts persist to disk so an 18K-token conversation resumes in 2.7s instead of 16.8s; draft depth 5 beats 4 (code 95 vs 89 tok/s). First long prompt after startup costs 6s extra (likely cold Engram reads from NVMe), and 32K prefill runs 30-37% below the published recipe numbers (1,707 vs 2,421 tok/s, cause unknown).
More from Infra
- SlateDB+Lance+Datafusion stack saves customer millions, as AI agents unbundle big data systems — sh_reya · 2026-10-18
- Paying $0.50 per 10 seconds for video gen APIs — when is self-hosting worth it? — Key-Lychee-913 · 2026-10-18
- vLLM and NVIDIA push Kimi K3 decode to 842 tokens/s on GB300 with fused CuTe DSL path — vllm_project · 2026-10-18
- SemiAnalysis's 1.68x vLLM speedup claim for Kimi K3 questioned over TP16 vs TP8 baseline — JiaZhihao · 2026-10-18
- Blackstone's AI empire: $200B data centers, $95B chip financing, bets on Anthropic and OpenAI — SuB8u · 2026-10-18
- Fix NVIDIA RTX VFX extension install failures in ComfyUI with one pip command — Mad4reds · 2026-10-18