tandem.cpp hits 97 tok/s on Qwen3.8-Flash-Next with one 4090, 3.3x faster than llama.cpp
smflx · reddit · 2026-11-02
An independent llama.cpp patch set, tandem.cpp, speeds up MoE models that don't fit in VRAM by keeping hot experts on GPU and offloading the rest to CPU, targeting single-user decode.
Results (i5-12400F + 64GB DDR4 + RTX 4090, GSM8K decode tok/s):
- Qwen3.8-Flash-Next IQ3S + MTP: 29.8 with llama.cpp → 97.3 (3.3x), close to Strata's 103.6
- gpt-oss-120b: 27.8 → 51.5 (1.85x)
- Other MoE models (GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1) typically gain 1.4-2.1x
- 64k prompt processing is 1.4-1.8x faster
Tuned for Qwen3.8 but works with any MoE model llama.cpp supports. Independent project, not affiliated with ggml-org; full results on 4 machines and build instructions on GitHub.
More from Infra
- $1T+ US Mega-Project Pipeline: Texas Eyed as the Future Hub for AI Intelligence Deployment — NinaDSchick · 2026-11-02
- Infra Play: 1GW of compute pays back in ~1 year as AI datacenter buildout rolls on — thedealdirector · 2026-11-02
- Musk says robotics needs more DRAM than Earth produces, yet Micron trades at 4x forward PE — sudoraohacker · 2026-11-02
- MiniMax thanks community quantizers running its 428B M3.1 Flash locally at 143.9GB — MiniMax_AI · 2026-11-02
- ML veteran Chris Albon jokes RAM prices are so high he'll save up 'from my cold dead hands' — chrisalbon · 2026-11-02
- Rebuilding 4 ComfyUI workflows into a zero-dependency app with DLSS 5 and VRAM streaming — MCP1964 · 2026-11-02