RunNburn: Open-Source Engine Runs 295B MoE on 64GB RAM Desktop, Beats llama.cpp
coderredlab · hn · 2026-07-30
RunNburn is an open-source Rust-based GGUF inference engine designed to run oversized models on consumer hardware that exceed VRAM/RAM capacities.
Core Mechanics:
- Weights are fully file-backed (mmap) with an explicit host memory budget (--ram-budget); GPU caches are dynamically sized based on actual free VRAM.
- No conversion steps or silent requantization—the GGUF file on disk is the single source of truth.
- Utilizes ODIRECT reads through iouring to batch-pull selected experts on demand.
Benchmark Results:
- Successfully ran Tencent's Hy3 (295B total / 21B active, 97.8 GiB Q2K GGUF) on a desktop with 64GB RAM and a single consumer NVIDIA GPU.
- Despite the file exceeding combined RAM and VRAM, warm-run decoding achieved 5.5 tok/s vs. 2.0 tok/s for llama.cpp.
Scope:: For models fitting comfortably in VRAM, llama.cpp remains faster due to years of kernel tuning. RunNburn targets models that "don't fit" and currently does not support continuous batching or multi-tenant throughput.
More from Infra
- MiniCPM-V 4.5 MoE reportedly hits 13+ tokens/s on iPhone with just 3GB memory — heyshrutimishra · 2026-10-16
- Solo-built Vulkan inference engine LaForge hits ComfyUI-level SD1.5 speed: 512px in 0.8s, 402MB container — Ichidan256 · 2026-10-16
- Trading 2 RTX 5090s for 4 R9700s: does doubled VRAM pay off? — poofph · 2026-10-16
- Running PolyStrata + Cline coding agent on a $0-fancy laptop at ~90 tok/s — rpwoerk · 2026-10-16
- TensorFold 1.0.12 runs 120B vision model on 1-2 DGX Sparks, adds Mac image input — HankYeomans · 2026-10-16
- vLLM brings frontier model local deployment to NVIDIA RTX Spark PCs — vllm_project · 2026-10-16