RustingStrata hits 160 tok/s on RTX 3060 running 35B MoE, ~2x faster than llama.cpp
Serious_Scarcity7818 · reddit · 2026-11-08
A developer's pure-Rust inference engine RustingStrata decodes Qwen3.6-35B-A3B (Q2KXL, 11.7 GiB, some experts on CPU) on a 12 GB RTX 3060:
- 118 tok/s decode with short prompts, 113 tok/s after a 4K prompt — 1.7x llama.cpp (68/66 tok/s)
- With new MTP (multi-token prediction) speculative decoding: up to 160 tok/s (152 after 4K)
- MTP gave no gains in llama.cpp on this card — the MTP layer eats VRAM, pushing more experts to CPU; some settings even crashed
- Prefill 2200 tok/s at 4K, on par with llama.cpp; 32K context still does 1800 prefill / 92 decode
- Vulkan matches CUDA speed, so AMD GPUs work; output matches llama.cpp top-1 98-99% of the time
- Single Rust binary, OpenAI-compatible server and chat UI; run with --gpu --mtp 1
Caveats: 160 tok/s is greedy decoding; MTP gains are larger for code and predictable text. Code is on GitHub, feedback from AMD users welcome.
Related event: Pure-Rust inference engine doubles llama.cpp speed on aging GPUs(2 posts)→
More from coding & agent
- Harrison Chase on the unsolved problems of always-on agents: notifications and memory — hwchase17 · 2026-11-08
- book-to-skill: an open-source pipeline that turns any book into an agent skill — adnan_hashmi · 2026-11-08
- Baseten Loops goes GA with serverless SFT/RL training, per-token pricing — baseten · 2026-11-08
- Software engineer asks how to bring AI agents into his family's retail business — FIREATWlLL · 2026-11-08
- Where do your agents run when the laptop closes? 4 hosting options compared — michaelTM_ai · 2026-11-08
- Where do browser agents actually break on the real web? Login walls, popups, or worse — lucy_yarntra007 · 2026-11-08