RustingStrata 1.3.0: a Rust GGUF engine that runs models bigger than your VRAM
Serious_Scarcity7818 · reddit · 2026-11-23
A developer released RustingStrata 1.3.0, a Rust-based GGUF inference engine designed to run models larger than available VRAM by keeping the most-used experts on GPU and offloading the rest to CPU.
- New interactive rustingstrata interface: scans its own folder, the Hugging Face cache and the llama.cpp cache, then walks you through mode (terminal chat, browser page, OpenAI-compatible API server, speed test) and three settings, printing the full command it will run.
- Other 1.3.0 changes: one server's chat page can load Inkling 2, Hy4-flash, GLM, DeepSeek and Gemma 4; Q40 models now take roughly half the VRAM; --auto-unload frees idle models.
- Prebuilt Linux/Windows binaries available (CUDA on NVIDIA, Vulkan on AMD/Intel), benchmarks in docs/BENCHMARKS.md.
More from Infra
- Apple's appenz: agent swarms are filling in NVIDIA's CUDA moat — appenz · 2026-11-23
- Tobi Lütke on Jensen: Nvidia's product 'takes rocks and makes them think' — MartinGTobias · 2026-11-23
- $200 Google AI Ultra subscription quietly includes a 96GB Blackwell GPU for 10 days — kieranklaassen · 2026-11-23
- Tesla's AI5 chip, 50X faster than AI4, goes to Optimus robots first while cars stay on AI4 — markjeffrey · 2026-11-23
- KV cache offload to RAM runs Qwen 3.8 with 8x200k streams on 4x3090 — Mr_Moonsilver · 2026-11-23
- OpenAI chart reveals unreleased GPT-6 Astra models; Domingos questions billions for <0.3 bits — pmddomingos · 2026-11-23