TensorSharp vs. llama.cpp: Benchmarking Muse Glimmer 30B Locally
fuzhongkai · reddit · 2026-08-14
A developer benchmarked Meta's Muse Glimmer 30B GGUF model using the open-source engine TensorSharp on an NVIDIA RTX PRO 6000 Blackwell GPU, comparing it directly against llama.cpp.
Key Findings:
- Plain Text Generation: TensorSharp leads in prefill speed by 16%-27% for short contexts (60-2050 tokens). However, llama.cpp regains the lead in both prefill and decode by 5%-14% for long contexts (>16k tokens).
- Speculative Decoding (DFlash): TensorSharp shows massive speedups (up to 164.6 tok/s) for mid-length prompts, though llama.cpp still dominates in long-context scenarios.
- Multi-GPU Tensor Parallelism: Using 2× RTX PRO 4000 GPUs (TP=2) yields significant 1.34x and 1.57x speedups for prefill and decode, respectively.
TensorSharp is a local GGUF inference engine supporting CUDA, Vulkan, Metal, and speculative decoding.
Related event: TensorSharp Outperforms llama.cpp in Local Inference Benchmarks(2 posts)→
More from Infra
- NVIDIA's NeMo Switchyard: Model Routing as the Agent Budget Manager — krishnan · 2026-08-14
- Running MiniMax H3 on RTX 5060 Ti: Resolution is the Real Bottleneck — danielcar · 2026-08-14
- RTX Pro 6000 Price Hikes: Buy a Workstation, Get the Rest for Free — Mr_Moonsilver · 2026-08-14
- NVIDIA Sol Engine Accelerates LTX-2.5 Video Generation by up to 4.68x — gan_chuang · 2026-08-14
- Running 2.4T Qwen3.8 Model on RTX 5090 + 5060 Ti: 0.8 tok/s Tested — mossy_troll_84 · 2026-08-14
- Cerebras and Cisco Shares Plunge Despite Strong Earnings Amid AI Supply Chain Bottlenecks — TiernanRayTech · 2026-08-14