tandem.cpp v0.3.0: 2-6x faster prefill for models too big for VRAM, Maya DeepSeek-V4.1 support
smflx · reddit · 2026-11-16
tandem.cpp v0.3.0 targets models much larger than VRAM+RAM: with expert cache, prompts run in 4096-token batches that borrow VRAM temporarily, no flags needed. Benchmarks on RTX 5070 Ti 16GB: 5.5k-token prompt drops from 28.9s (llama.cpp) to 4.1–7.0s; DeepSeek-V4.1 IQ2XS 15.7k prompt from 2012s to 195s. Against Project Maya on the same GGUFs, tandem.cpp decodes 1.5x faster but Maya leads on the longest prompts. Caveats: Linux only, ROCm lacks VRAM borrowing, huge models still hit SSD.
Related event: tandem.cpp v0.3.0 Boosts Large Model Prefill Speed 2-6x(2 posts)→
More from Infra
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- Qdrant 1.20.1: 50% fewer cold-storage reads, ~2x faster quantization, agent-ready setup — qdrant_engine · 2026-11-16
- One 16GB GPU runs songs, voice, images, 720p video and local RAG — all offline and open source — llama-of-death · 2026-11-16
- Starlink Now Protects Greek Trains From Collisions With 5cm-Accurate Tracking of 134 Locomotives — XFreeze · 2026-11-16
- Google donates Agent Substrate, gVisor and OpenRL to CNCF; Agent Substrate 1.0 goes GA on GKE — dl_weekly · 2026-11-16
- Nvidia scandal isn't slowing down: Huang's near-$100B power web under strain — The Verge AI · 2026-11-16