A llama.cpp VRAM-cache trick hits 340 pp/s on Kimi K2.7 Code
ylchao · reddit · 2026-07-22
The post benchmarks a llama.cpp trick that uses VRAM as a cache layer for MoE models, keeping expert weights on the CUDA path and avoiding some full-memory pressure.
Key results reported:
- On a DGX Spark running Kimi K2.7 Code GGUF, the best setup reached 340 pp/s prefill and 9.6 tg/s decode.
- The author compares several offload regex and unified-memory configurations, showing that pinning and batch settings materially affect throughput.
- On 3090-class hardware, the same idea was tested on a MiniMax-M2.7 GGUF model, with different tradeoffs between prefill speed and token throughput.
The takeaway is that MoE models can run faster than expected on constrained hardware if the active experts stay on the GPU path while the rest is treated as cache/storage, and the post includes concrete llama-bench flags for reproduction.
More from Infra
- OpenFPM CUDA-style kernels now run on Apple Silicon GPUs via Metal — Scobleizer · 2026-07-22
- SkewAdam cuts MoE optimizer memory by 97.4% and fits 6.78B on a 40GB GPU — Kooky-Ad-4124 · 2026-07-22
- Samsung is said to weigh a €1B Mistral investment at a €20B valuation — rohanpaul_ai · 2026-07-22
- NVIDIA and ETH Zürich cut small-message AllReduce latency by deleting barriers — thoefler · 2026-07-22
- Grok Build adds token usage, batching and diagnostics for developers — elonmusk · 2026-07-22
- Qwen3.8-Max-Preview ranks No. 1 on NVIDIA’s FlashInfer benchmark — Scobleizer · 2026-07-22