TensorSharp vs. llama.cpp: New Open-Source Inference Engine Shows Strong Local Performance
fuzhongkai · reddit · 2026-08-14
A developer published a comprehensive benchmark comparison between the open-source inference engine TensorSharp and llama.cpp.
- Setup: Tested Meta's Muse Glimmer 30B GGUF model on single and dual NVIDIA RTX PRO 6000/4000 Blackwell GPUs.
- Plain Text: TensorSharp outperforms llama.cpp in prefill speed for shorter contexts (up to 1.27x faster), though decode speeds are mixed. llama.cpp dominates overall throughput in very long contexts (>16k tokens).
- Speculative Decoding: Using DFlash draft models, TensorSharp excels in mid-length contexts, while llama.cpp delivers higher and more stable decode speeds for very long contexts.
- Multi-GPU: TensorSharp demonstrates strong scaling on dual GPUs with TP=2, achieving 1.34x prefill and 1.57x decode speedups.
TensorSharp is positioned as a highly optimized local inference engine supporting continuous batching, multimodal, and speculative decoding.
Related event: TensorSharp Outperforms llama.cpp in Local Inference Benchmarks(2 posts)→
More from Infra
- Databricks Introduces Smart Routing in Unity AI Gateway, Claims 30%+ Cost Reduction — matei_zaharia · 2026-08-14
- OpenAI Acquired 4.2% Stake in Cerebras Before Ultrafast Launch — ryanmerket · 2026-08-14
- Jensen Huang Marks DGX 10th Anniversary, Unveils DGX Spark at 5x Original Power — nvidia · 2026-08-14
- NVIDIA x Runway: Gen-4.5 Integrated into Vera Rubin Platform in One Day — nvidia · 2026-08-14
- RTX 5090 Test: SageAttention Nearly Doubles Video Generation Speed — gabxav · 2026-08-14
- CoreWeave Sandbox Lets You Drive Claude Agents From Your Phone — wandb · 2026-08-14