TensorSharp vs llama.cpp: audited benchmarks show up to 5.5x prefill gains on A40s, with caveats
fuzhongkai · reddit · 2026-10-25
A r/LocalLLaMA post carefully separates what TensorSharp — a native .NET 10 GGUF inference engine by zhongkaifu — documents in its repo from what readers can reproduce, with extensive caveats:
- Datacenter A40s: DeepSeek V4.1 Flash on 5×46GB A40s, 32K context: exact prefill 563.9 vs llama.cpp 174.6 (3.23x); approximate YOCO mode 960.5 (5.50x, valid only for that 1.8K prefill cell). Muse Spark 816B on 10×A40s: prefill 1.63-2.07x, decode 1.40x, layer-split decode 0.99x — slightly slower, reported deliberately.
- CPU offload: on a 2×RTX PRO 6000 Blackwell host, Gemma 4 26B-A4B runs in 10.8GiB with all experts in system RAM; across three MoE models, prefill 4.5-10.9x and decode 2.5-4.5x faster at 1.0-3.5x the VRAM. DeepSeek V4 offloaded decode is the case where llama.cpp wins.
- Apple M5 Pro: absolute numbers only, near parity (<1.0x), confounded by different ggml checkouts.
- Laptop-GPU speculative decoding: RTX 3080 Laptop + Qwen3.8-27B, DFlash2 reaches 1.63x (31.7 tok/s) on predictable prompts.
The docs explicitly warn against merging speedups across hardware/scenarios into one marketing number.
Related event: TensorSharp Claims Up to 5.5x llama.cpp Prefill Throughput on DeepSeek(3 posts)→
More from Infra
- Samsung reportedly using TSMC 3nm for Nvidia's next-gen NVHBM custom chip base dies — Beth_Kindig · 2026-10-25
- No AI bubble yet: revenue per GPU minus rental is the real equation — PTrubey · 2026-10-25
- AI labs are flipping GPU contracts: H100/B200 rental prices up 2-3x since early lock-ins — Yuchenj_UW · 2026-10-25
- Swarms open-sources RouteHub, an LLM gateway that imports 190x faster than litellm — KyeGomezB · 2026-10-25
- Radeon 9700 Prices Double in a Month, Hurting Personal Inference Server Builds — UltramanQuar · 2026-10-25
- New blog demystifies GPU communications, from page tables to NVLink, via SGLang internals — BanghuaZ · 2026-10-25