RustingStrata runs a 177B model on a 12GB GPU at 21 tok/s, up to 2x llama.cpp decode
Serious_Scarcity7818 · reddit · 2026-11-06
The author benchmarked RustingStrata vs. llama.cpp back-to-back on every Qwen GGUF on disk (RTX 3060 12GB, Ryzen 5 7600X, 32GB RAM, browser and Discord open, mean of 2 runs).
Findings:
- Small dense models: llama.cpp still prefills faster (e.g., 2B: 6080 vs. 3859 tok/s); decode is mixed
- Once models exceed VRAM (35B/80B/177B on a 12GB card), Strata wins 1.5x prefill and up to 2x decode:
- Qwen3.6-35B-A3B: 1539 vs. 1029 prefill, 71.5 vs. 39.6 decode
- Qwen3-Next-80B-A3B: 63.7 vs. 33.3 decode
- 177B Flash-Next: 305 vs. 193 prefill, 21.3 tok/s decode
- 80B hits 64 tok/s decode on a 3060; MTP on pushes 35B to 160 tok/s
- Hand-tuned llama.cpp (-ncmoe 12) still beats Strata on 35B (2236 prefill / 68 decode)
Full benchmark commands for both engines are provided, making the comparison reproducible.
Related event: RustingStrata: Pure Rust Inference Engine Runs Qwen MoE on Consumer GPUs(3 posts)→
More from Infra
- Blogger Proposes Data Centers Win Locals Over With Free ChatGPT Pro and Waste Heat — flowersslop · 2026-11-06
- Voters reject data centers in 40 of 41 ballot measures; Musk eyes orbital data centers — PeterDiamandis · 2026-11-06
- Ripple-65B open-weight model runs at 7 tok/s locally on a 64GB Mac mini — DZhang50 · 2026-11-06
- Liquid AI: a 9B LLM can't run in a car, but LFMs can — JosephJacks_ · 2026-11-06
- Nvidia chips pile up in Hong Kong as China withholds import approval — pstAsiatech · 2026-11-06
- Chart: UK IT investment at 0.66% of GDP vs US 2.21%, missing the AI capex boom — KeeganMcB · 2026-11-06