RustingStrata benchmarked 1.29x faster than llama.cpp with no measurable quality loss on RTX 3060
Serious_Scarcity7818 · reddit · 2026-11-11
A Reddit user compared the pure-Rust MoE inference engine RustingStrata against llama.cpp using the same Qwen GGUF IQ2XS quant, identical prompt, greedy decoding, and 128 tokens on an RTX 3060 12GB:
- Generation: 33.0 vs 25.6 tok/s (1.29x), range over 6 runs 29.3–35.8 vs 23.2–26.9; prefill 109 vs 31 tok/s — nearly 3.5x faster (the project's image claims 2x prefill for Qwen3.6-35B-A3B on a 12GB GPU).
- Quality check on the same 9,180 WikiText-2 tokens: perplexity 2.9212 vs 2.9156, 96.78% top-token agreement, mean KLD 0.0098 — differences within run-to-run noise.
Conclusion: on this model/quant/GPU, speed wins with no measurable quality loss; the author explicitly avoids generalizing.
More from Infra
- Batch APIs: the 50% LLM discount only pays off after model choice and prompt caching — arpit_bhayani · 2026-11-13
- RTX Pro 6000 bricks mid-Qwen run, owner jokes it "died of a broken heart" over Jensen news — wine_dark · 2026-11-13
- Bittensor's Chutes Handles 256 Concurrent Requests on a Single RTX 5090 at ~18k tok/s — markjeffrey · 2026-11-13
- Google expands TPU v8 lineup: 8tx cuts latency 2x, 8i Max doubles silicon per rack — Beth_Kindig · 2026-11-13
- Horace He to share two mental models for reasoning about inference at SGLang Summit — cHHillee · 2026-11-13
- Used A100 prices fall another 20-30% as sellers rush ahead of a collapse — Bascilian · 2026-11-13