RunNburn: Open-Source Engine Runs 295B MoE on 64GB RAM Desktop, Beats llama.cpp

coderredlab · hn · 2026-07-30

RunNburn is an open-source Rust-based GGUF inference engine designed to run oversized models on consumer hardware that exceed VRAM/RAM capacities.

Core Mechanics:

Weights are fully file-backed (mmap) with an explicit host memory budget (--ram-budget); GPU caches are dynamically sized based on actual free VRAM.
No conversion steps or silent requantization—the GGUF file on disk is the single source of truth.
Utilizes ODIRECT reads through iouring to batch-pull selected experts on demand.

Benchmark Results:

Successfully ran Tencent's Hy3 (295B total / 21B active, 97.8 GiB Q2K GGUF) on a desktop with 64GB RAM and a single consumer NVIDIA GPU.
Despite the file exceeding combined RAM and VRAM, warm-run decoding achieved 5.5 tok/s vs. 2.0 tok/s for llama.cpp.

Scope:: For models fitting comfortably in VRAM, llama.cpp remains faster due to years of kernel tuning. RunNburn targets models that "don't fit" and currently does not support continuous batching or multi-tenant throughput.

Original post →

RunNburn: Open-Source Engine Runs 295B MoE on 64GB RAM Desktop, Beats llama.cpp

More from Infra