Tandem-vLLM 0.2.0 runs DeepSeek-V4.1-Flash on 3090/4090-class GPUs with 2-bit expert offload
smflx · reddit · 2026-11-14
Tandem-vLLM is an expert-offload fork of vLLM 0.31 for DeepSeek-V4.1-Flash (293 GB package). Version 0.2.0 adds Hopper, Ada and Ampere support plus two-GPU modes.
- Routed MoE experts live in host RAM as 2-bit vector codes (2.1 bits/weight); cache misses are computed by the CPU in parallel with the GPU, with exact outputs.
- Dual-GPU modes: tensor parallel (best with strong CPUs, 93.1 tok/s on 2× RTX PRO 6000) and "helper GPU" (second card handles hot experts, better with weak CPUs, 21.7 tok/s on 2× 24 GB).
- Quality vs official FP4: GPQA-diamond drops 81.3%→73.2% with pure VQ; FP4TOP=52 recovers about half. GSM8K and long-context retrieval largely unaffected.
- Requirements: 300 GB disk, 128 GB+ RAM, Linux, CUDA 13. A llama.cpp-based sister project tandem.cpp hits 33 tok/s on a 5090 + 128 GB RAM.
More from Infra
- Jensen Huang's exit could trigger a credit crunch across Wall Street's $500B AI financing bet — minhnguyen · 2026-11-14
- Quantizing training to MXFP8 with stochastic rounding: the accuracy-speed sweet spot — snowclipsed · 2026-11-14
- NVIDIA lands nvfp4 in large-scale pretraining under codename "yolo bundle" — SonglinYang4 · 2026-11-14
- Drogue raises $20M to power floating data centers with the ocean's heat — arian_ghashghai · 2026-11-14
- Megaspeed was NVIDIA's largest chip buyer in Southeast Asia by 2025, now official cloud partner — RileyRalmuto · 2026-11-14
- AWS adds OpenFold3 to SageMaker JumpStart for private biomolecular structure prediction — AWS ML Blog · 2026-11-14