Open-source Strata runs 125B MoE models on 12GB VRAM at up to 93 tokens/s
udmrzn · x · 2026-10-03
The open-source inference engine Strata runs 125B-class MoE models (Qwen3.8-Flash-Next) locally on consumer GPUs like an RTX 5070 12GB by splitting work across GPU, CPU, RAM, and SSD.
Key mechanisms:
- Doesn't load the whole model into VRAM: GPU holds Attention, Router, KV Cache, and hot experts; cold experts stay in RAM (CPU compute) with large lookup tables on SSD read on demand
- Adaptive expert caching lets 12–16GB VRAM handle huge MoE models
- Speculative decoding via the model's MTP layers, claimed 1.6–1.8x speedup
Benchmarks (RTX 5070 12GB + Ryzen 5 7600 + 64GB RAM, short context): 93 tokens/s at Q20, 79 at IQ2XS; at 128K context still 46–74 tokens/s. Recommended: 12GB+ VRAM, 64GB RAM, NVMe SSD, 70–120GB storage.
More from Infra
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19