RustingStrata: pure-Rust inference engine doubles Qwen MoE speed on a 12 GB GPU
Serious_Scarcity7818 · reddit · 2026-11-05
A developer released RustingStrata, a pure-Rust (no ggml) inference engine for Qwen GGUF models. CUDA kernels are compiled at startup via NVRTC and the same kernels run on Vulkan.
Key trick: for MoE models that don't fit in VRAM, overflow experts run on CPU concurrently with GPU work while hot experts are continuously migrated to the GPU.
Benchmarks (RTX 3060 12 GB):
- Qwen3.8-Flash-Next (IQ2XS): 36 tok/s decode, 370 tok/s prefill vs original Strata's 19-22 / 140-146, with token-identical greedy output
- Qwen3.6-35B-A3B Q2KXL (doesn't fit): 118 tok/s decode vs 68 for a heavily tuned llama.cpp
- Qwen3-Next-80B IQ1S with most experts on CPU: 92 tok/s (llama.cpp 44)
- AMD RX 7900 XT (Vulkan/RADV): 209 tok/s on 35B-A3B, ahead of llama.cpp Vulkan/ROCm
Quant support spans IQ1S to Q80 plus BF16/F16; ships an OpenAI-compatible server with tool calling and a self-tuning calibrate command. Honest caveat: cold-cache first-prompt prefill is disk-bound, and prefill roughly ties llama.cpp.
More from Infra
- LFM3-Omni-2B ported to run fully offline on iPhone at ~80 tok/s with custom WebGPU kernels — JosephJacks_ · 2026-11-05
- Orbital Data Centers Launch to Sun-Synchronous Orbit, Memed as Elves Sailing West — rand_longevity · 2026-11-05
- Vietnam's Lâm Đồng to bring Starlink to mountain areas where poor signal hampers rescue — elonmusk · 2026-11-05
- SpaceX partners with Qualcomm for nationwide Starlink Mobile 5G network from 2027 — Polymarket · 2026-11-05
- Stopping runaway agent spend before the bill: pre-call guardrails and three gaps found the hard way — MaverikSh · 2026-11-05
- US AI scale faces two physical gates: delivered power and manufacturing — ingliguori · 2026-11-05