Open-source Rust GGUF runtime runNburn runs 295B model on 64GB RAM, 2.8x faster decode than llama.cpp
coderyeon · reddit · 2026-07-31
Developer coderyeon shares runNburn, an open-source Rust inference runtime for GGUF models designed for memory-constrained environments. It supports CPU, CUDA, Metal, and Android, with experimental Vulkan and OpenCL. Key feature is explicit memory budgeting: file-backed weights, bounded host residency, and accelerator caches sized to hardware, without requantization or conversion.
Performance highlights:
- Desktop: On a 64GB RAM + RTX 3090, runs Tencent's Hy3 295B-A21B (97.8 GiB Q2K/Q3K GGUF) at 5.513 tok/s short-context decode, 2.8x faster than llama.cpp, and 7.3x faster prefill; long context still page-fault bound.
- Mobile: On 8GB Galaxy Z Flip4, runs Qwen3.6 35B-A3B with 2.11x faster end-to-end time than MNN (63.2s vs 133.5s), and 2.33x faster decode.
Project started in March 2026 for mobile, now supports multiple architectures (dense attention, GatedDeltaNet, Mamba, sparse MoE). Target is personal single-owner inference server, not high-throughput multi-tenant.
More from Infra
- Revisiting Lossy Verification in Speculative Decoding: Mechanisms and Failure Modes — Tianyu Wang · 2026-07-31
- Energy Consumption: Single AI Prompt vs Agentic Workflow Differs by 100,000x — AndyMasley · 2026-07-31
- UBS Chart Highlights the Central Hub of the AI Compute Supply Chain — BenBajarin · 2026-07-31
- Stacking 512GB VRAM: Developer Builds Dual-Node 8x V100 Inference Cluster — UltraFOV · 2026-07-31
- Amazon Says AI Demand Is in Early Stages, Projects Over $220B CapEx for 2026 — inductionheads · 2026-07-31
- Stanford Paper Proposes Mandatory Resource Disclosure for Hyperscale Data Centers — ruthstarkman · 2026-07-31