Pure-Rust engine runs 249GB DeepSeek model on a single 12GB RTX 3060
Serious_Scarcity7818 · reddit · 2026-11-10
RustingStrata, a pure-Rust GGUF inference engine with CUDA and Vulkan backends, now runs DeepSeek-V4.1-Flash IQ2XXS (249GB) on a single RTX 3060 12GB + 30GB RAM: experts stream from SSD through the page cache, with a 9GB LRU expert cache in VRAM; attention stays on GPU.
vs llama.cpp b12116 (--cpu-moe) on the same box: decode 2-3.6 tok/s vs 0.3-1.5 tok/s, faster prefill too; top-1 agreement 189/192 on short prompts under teacher forcing. Caveats: greedy decoding only, 4096 context cap, no sampling/tool calls yet, and speed varies with page cache since the model far exceeds RAM.
More from Infra
- Texas A&M Builds One of the Strongest US Academic Supercomputers, Hiring 10 — yuz9yuz · 2026-11-13
- Tencent Reportedly DeepSeek's Biggest Backer as China Compute Centralizes — teortaxesTex · 2026-11-13
- Data Architecture 101: one infographic covering terms, models and patterns — nevrekaraishwa2 · 2026-11-13
- Rumors swirl that Azure runs on shaky foundations as major outages hit with no root cause — thedealdirector · 2026-11-13
- NAVER AI Lab's FOAM optimizer halves optimizer state while matching SOAP and Muon — naver-ai · 2026-11-13
- Microsoft outage disrupts LA County Sheriff's email and communication systems — Kyrannio · 2026-11-13