tandem.cpp v0.3.0: 2-6x Faster Prompt Processing, DeepSeek-V4.1 GGUFs Supported
smflx · reddit · 2026-11-16
Local inference engine tandem.cpp ships v0.3.0 (first tagged release), targeting speedups when the model far exceeds VRAM+RAM (e.g., 16GB card + 32GB RAM running large MoEs).
Headline: with the expert cache on, prompts run in 4096-token batches — 2-6× faster prompt processing. On an RTX 5070 Ti, Qwen3.8 5.5k-token prompts drop from 12.4s to 4.1-7.0s; DeepSeek-V4.1 IQ2XS 15.7k prompts fall from llama.cpp's 2012s to 195s. Decode also leads clearly (110 tok/s vs llama.cpp's 27 after a 5.5k prompt).
vs. Project Maya: on the same DeepSeek-V4.1 Maya-XS GGUF, tandem.cpp decodes at 60 tok/s without drafting vs Maya's 39-40 (1.5×), but Maya with its DSpark drafter is still faster (54.9 tok/s), and Maya leads on the longest prompts — the next work item.
Caveats: Linux only; AMD works but VRAM borrowing is off; oversized models still hit the SSD on long prompts; Windows builds pending.
More from Infra
- Rumor: DeepSeek V5 in pretraining, ~2T-param sparse MoE trained entirely on Huawei Ascend, no NVIDIA — DeryaTR_ · 2026-11-16
- Microsoft email outage delayed chlorine alerts at German theme park hotel, 15 guests harmed — evilsocket · 2026-11-16
- How Lisa Su took AMD from near-bankruptcy to $1T — and she's Jensen Huang's cousin — bigaiguy · 2026-11-16
- llmfit scans your hardware and tells you exactly which local LLMs will actually run — thisdudelikesAI · 2026-11-16
- SemiAnalysis reportedly cuts 2028 AI accelerator capex forecast by 28% — firstadopter · 2026-11-16
- UW's Mobile AI Factory Auto-Builds Phone-Specific Small Language Models — UW · 2026-11-16