TensorSharp MoE Offload Slashes VRAM Use, Outperforms llama.cpp by up to 8x

fuzhongkai · reddit · 2026-08-05

TensorSharp has merged a new MoE CPU-offload feature into its main branch, allowing routed expert weights to be kept in system RAM. This enables running large MoE models (like a 35B parameter model) on consumer GPUs with only 12-16GB of VRAM alongside long-context KV caches.

A comprehensive benchmark against llama.cpp was conducted on a server with dual RTX PRO 6000 GPUs. When CPU offloading is enabled, TensorSharp drastically outperforms llama.cpp:

Qwen 3.5 35B: With 48 layers offloaded, TensorSharp achieves up to 8.85x faster prompt processing (pp4096) and 3.31x faster generation (tg128).
Gemma 4 26B: Full offload yields up to 5.59x faster prompt processing and 2.93x faster generation.
GPT-OSS 20B: Offloading 12 layers results in 5.50x faster prompt processing and 2.83x faster generation.

TensorSharp does consume more VRAM than llama.cpp when offloading, but the massive speedups in prompt processing and token generation make it a highly efficient tool for local deployment of large models.

Original post →

TensorSharp MoE Offload Slashes VRAM Use, Outperforms llama.cpp by up to 8x

More from coding & agent