Pure-Rust Inference Engine RustingStrata Doubles llama.cpp Prefill Speed for Qwen3.6-35B on a 12GB GPU
Serious_Scarcity7818 · reddit · 2026-11-09
A developer benchmarked RustingStrata, a pure-Rust MoE inference engine, against llama.cpp running unsloth's Qwen3.6-35B-A3B UD-Q5KM (MTP version, 2/3 experts on CPU) on an RTX 3060 12GB + Ryzen 5 7600X + 32GB DDR5:
- Decode, short prompt: 70 tok/s vs 43; after a 4K prompt: 65 vs 43
- Prefill at 4K: 1870 vs 1210 (roughly 2x)
- 32K context: 1470 prefill / 49 decode vs 1030 / 29
- With MTP enabled on both: 62 vs 45-56 tok/s; llama.cpp MTP shows 63% draft acceptance and +25% decode
- Vulkan backend decodes at 60 (85 with MTP) with slower prefill than CUDA
KV cache is f16. Code is open source on GitHub; the author is soliciting feedback from AMD users.
More from Infra
- Texas A&M Builds One of the Strongest US Academic Supercomputers, Hiring 10 — yuz9yuz · 2026-11-13
- Tencent Reportedly DeepSeek's Biggest Backer as China Compute Centralizes — teortaxesTex · 2026-11-13
- Data Architecture 101: one infographic covering terms, models and patterns — nevrekaraishwa2 · 2026-11-13
- Rumors swirl that Azure runs on shaky foundations as major outages hit with no root cause — thedealdirector · 2026-11-13
- NAVER AI Lab's FOAM optimizer halves optimizer state while matching SOAP and Muon — naver-ai · 2026-11-13
- Microsoft outage disrupts LA County Sheriff's email and communication systems — Kyrannio · 2026-11-13