RustingStrata: run 100GB MoE models on consumer GPUs, 4x speedup with two cards
Serious_Scarcity7818 · reddit · 2026-11-06
RustingStrata is a Rust inference engine for running large Qwen MoE models on one consumer GPU plus system RAM, offloading experts that don't fit in VRAM to the CPU. Key updates:
- Correctness: a new perplexity command compatible with llama-perplexity shows strata is closer to llama.cpp CUDA (mean KLD 0.0024, 97.9% same top token on Qwen3-4B) than llama.cpp's own CPU build (0.0062).
- Multi-GPU: --device 0,1 splits layers by free VRAM across mixed cards/backends (CUDA+Vulkan). On 2× RTX 3090 running a 98GB Qwen3.8-Flash-Next: 70 tok/s decode vs 18–34 on one card, 1465 tok/s prefill at 4K. Caveat: if the model fits on the faster single card, splitting is slower; AMD cards also drop clocks during idle gaps (forcing high clocks: 44→57 tok/s).
- No terminal needed: strata serve ships a browser chat page with streaming, code blocks, thinking mode, HF GGUF downloads and hot model switching; on Windows, double-click the binary.
- Coding agents: speaks the Anthropic Messages API so Claude Code can point at it via ANTHROPICBASEURL; cached post-compaction state cuts compacted requests from 5.4s to 1.0s.
- More: sparse attention (QSA) for Qwen3.8-Flash-Next — 18.4 tok/s at 128K context on an RTX 3060 12GB; AMD fixes; Q40/Q50 quant support.
Related event: RustingStrata: pure-Rust engine runs large Qwen MoE models on consumer GPUs(2 posts)→
More from coding & agent
- Exo paper: agents that rewrite their own harness at runtime, cutting model cost up to 82% — sarahcat21 · 2026-11-06
- Study: LLMs Notice Context Conflicts 73% of the Time but Tell Users 0% — rseroter · 2026-11-06
- Ampersand matches frontier coding models at 70% lower cost via co-coding training — niloofar_mire · 2026-11-06
- Fo Friends launches: AI agents now talk to each other to plan dinners and gifts — heyshrutimishra · 2026-11-06
- AI coding chaos: 51 copies of Playwright across 12 versions found on one machine — julianharris · 2026-11-06
- Vercel launches Schedules public beta: agents can create cron jobs at runtime — cramforce · 2026-11-06