Betelgeuse runs 68GB Qwen MoE bit-exact at 22 tok/s on RTX 3060 + 16GB RAM
zyxciss · reddit · 2026-10-31
Reddit user zyxciss released Betelgeuse (bel-cli/bel-server), a fork of llama.cpp (with an upstream PR planned), that runs the 68GB Qwen 3.8 Flash Next GSQ-RCO-IQ2XS build at a sustained 22 tok/s (peaks of 48 tok/s) on an RTX 3060 12GB + 16GB DDR4 PC—100% bit-exact.
Motivation
- The author admits an earlier prototype cheated via gate pruning (dropping cold experts), which silently degraded output quality, and was scrapped.
- Stock llama.cpp thrashes mmap (1,500 major page faults/token) at 1.47 tok/s; Strata needs 48–64GB RAM for 40+ tok/s and drops to 8.11 tok/s on 16GB.
How it works (no experts dropped)
- Zero gate pruning, bit-exact: all 10 routed experts per layer across 48 layers computed at full precision; 28,080 router records and 576 generated tokens verified byte-for-byte (0.000e+00 diff).
- Arrival-Slack VRAM slot pool: dense layers + GDN + QSA + LM head fit in 3.2GB VRAM; remaining VRAM holds 5,000 expert slots distributed non-uniformly by per-layer compute slack (240 slots on layers 0–4, 92 mid, 60 tail).
- In-stream GPU doorbell + single CUDA Graph replay: a custom CUDA kernel resolves expert hits in 2µs on-device, replacing 48 graph splits/token (360ms in cudaStreamSynchronize); the whole decode pass is one CUDA Graph replay.
- RAM victim cache + iouring pager: evicted experts move to a dedicated host cache with no VRAM/RAM duplication; NVMe reads drop from 71 to 12.43 MB/token.
- No duplicated expert blobs on disk: 3.2GB Dense.gguf + 35.3GB experts.bin (4KB-aligned single-read blobs) + 26.8GB n-gram shard.
Tested on CachyOS/Arch inside a strict MemoryMax=12G cgroup, with extra RAM sticks physically removed to simulate a typical 16GB machine.
More from Infra
- 5 cheap open models vote vs Opus: 89.6% accuracy at 1/10th the cost — PomegranateAble4339 · 2026-10-31
- MiniMax 3 video generation demoed running locally on a single RTX 3090 Ti — Impossible-Ad-3798 · 2026-10-31
- 15-year-old builds sliding window attention from scratch: 4.16x speedup, 21% less VRAM — vankownlaw · 2026-10-31
- Microsoft offering 60,000 employees buyouts as it shifts spend from humans to datacenters — Polymarket · 2026-10-31
- Baseten set to raise $2B at up to $30B valuation as revenue triples — rohanpaul_ai · 2026-10-31
- ~25 Internet-Facing Palo Alto Firewalls Rooted One Week After php2shell Patch — cyb3rops · 2026-10-31