tandem.cpp doubles MoE offload speed: DeepSeek-V4.1 on a 16 GB GPU, now with ROCm
smflx · reddit · 2026-11-15
Author updated tandem.cpp, a llama.cpp patch set that keeps hot MoE experts on GPU and offloads the rest to CPU, now with AMD ROCm/HIP support.
DeepSeek-V4.1-Flash IQ2XS (139 GiB of experts, 64 GB RAM PCs, streaming from SSD):
- RTX 5070 Ti 16 GB + i5-14600KF: 7.4 tok/s on llama.cpp vs 14.2 on tandem.cpp (Strata: 10.4)
- With a DRAM-less budget SSD it still hits 10.4 tok/s (Strata: 5.7)
- RTX 5090 32 GB + 128 GB RAM: 11.5 → 33.1 tok/s
Less RAM than the model: on a 32 GB RAM machine, Qwen3.8-Flash-Next IQ3S (78 GiB) + MTP reaches 34.3 tok/s vs 14.9 on llama.cpp. When the model fits in RAM, Strata is slightly ahead (104 vs 97 on a 4090).
DeepSeek-V4.1, gpt-oss-120b, GLM-4.5-Air and Qwen3.6 all gain 1.4-2.1x over vanilla llama.cpp. On RX 7900 XTX tandem.cpp hits 68.6 tok/s, matching Strata (Avalon leads at 83.6); new shared CUDA/HIP kernels add +2.5% on the 7900 XTX and +1.3% on a 5070 Ti. Vulkan not yet supported.
More from Infra
- JP Morgan sees AI accelerator shipments doubling to 31.2M units by 2028, custom silicon leading — Beth_Kindig · 2026-11-15
- Home Lab: 8x RTX 3090 Rig Runs 10 Parallel Agents at 370 tok/s — dazzou5ouh · 2026-11-15
- Alchemy adds Rivet, Effect Cluster and 8 sandbox platforms with swap-one-layer portability — samgoodwin89 · 2026-11-15
- Claim: Gemini 4 Trained Entirely on Google's Own TPUs, No Nvidia Involved — gaganghotra_ · 2026-11-15
- Chamath explains why CoreWeave credit default swap spreads are jumping — rohanpaul_ai · 2026-11-15
- Solar-powered local AI: old MacBook + Meshtastic mesh keeps inference running off-grid — natesiggard · 2026-11-15