tandem.cpp v0.3.0: 2-6x Faster Prompt Processing, DeepSeek-V4.1 GGUFs Supported

smflx · reddit · 2026-11-16

Local inference engine tandem.cpp ships v0.3.0 (first tagged release), targeting speedups when the model far exceeds VRAM+RAM (e.g., 16GB card + 32GB RAM running large MoEs).

Headline: with the expert cache on, prompts run in 4096-token batches — 2-6× faster prompt processing. On an RTX 5070 Ti, Qwen3.8 5.5k-token prompts drop from 12.4s to 4.1-7.0s; DeepSeek-V4.1 IQ2XS 15.7k prompts fall from llama.cpp's 2012s to 195s. Decode also leads clearly (110 tok/s vs llama.cpp's 27 after a 5.5k prompt).

vs. Project Maya: on the same DeepSeek-V4.1 Maya-XS GGUF, tandem.cpp decodes at 60 tok/s without drafting vs Maya's 39-40 (1.5×), but Maya with its DSpark drafter is still faster (54.9 tok/s), and Maya leads on the longest prompts — the next work item.

Caveats: Linux only; AMD works but VRAM borrowing is off; oversized models still hit the SSD on long prompts; Windows builds pending.

Original post →

More from Infra

Infra channel →