tandem.cpp v0.3.0: 2-6x faster prefill for models too big for VRAM, Maya DeepSeek-V4.1 support

smflx · reddit · 2026-11-16

tandem.cpp v0.3.0 targets models much larger than VRAM+RAM: with expert cache, prompts run in 4096-token batches that borrow VRAM temporarily, no flags needed. Benchmarks on RTX 5070 Ti 16GB: 5.5k-token prompt drops from 28.9s (llama.cpp) to 4.1–7.0s; DeepSeek-V4.1 IQ2XS 15.7k prompt from 2012s to 195s. Against Project Maya on the same GGUFs, tandem.cpp decodes 1.5x faster but Maya leads on the longest prompts. Caveats: Linux only, ROCm lacks VRAM borrowing, huge models still hit SSD.

Related event: tandem.cpp v0.3.0 Boosts Large Model Prefill Speed 2-6x(2 posts)→

Original post →

More from Infra

Infra channel →