Betelgeuse runs 4-bit MoE at 24 tok/s on RTX 3060 with 16GB RAM, 100% byte-exact
zyxciss · reddit · 2026-11-01
The Betelgeuse author addressed two criticisms from a previous thread — that IQ2XS was too low-bit, and that testing on a single 16GB DDR4 stick was unrepresentative — by benchmarking the full 4-bit IQ4XS (100GB-class experts) on the same RTX 3060 12GB under both RAM configs with 16GB total.
Key results:
- IQ4XS + 2x8GB dual-channel: 24.0 tok/s (stock llama.cpp 0.6, Strata --low-ram 5-6)
- IQ4XS + 1x16GB single-channel: 14.0 tok/s
- Q8KXL: 8.0 cold / 12.0 warm tok/s (stock OOMs)
- Output is 100% byte-exact vs. stock in all configs, zero dropped experts
Why dual-channel doubled throughput: Betelgeuse uses an exclusive host-RAM victim cache plus async hardware DMA (MOEDMA=1) between the VRAM slot pool and NVMe. Single-channel DDR4 (17-19 GB/s) forces CPU and PCIe DMA to fight over one memory channel; doubling bandwidth lets the GPU's DMA engine pull victim-cache hits at full speed, jumping 14 → 24 tok/s.
Engine on GitHub, pre-packed bundles on HuggingFace, and an upstream llama.cpp PR are live. Next: porting the same streaming + victim-cache architecture to DeepSeek V4.1 Flash, already hitting 25 tok/s on similar hardware in early testing.
Related event: Betelgeuse runs 68GB Qwen MoE on RTX 3060, benchmark data released(2 posts)→
More from Infra
- Musk: Colossus 2 to host ~1.2M Blackwell GPUs by end of year — scaling01 · 2026-11-01
- Tokenomos open-sourced: rewritten vLLM frontend makes small-model inference ~3x faster on RTX 5090 — bdsqlsz · 2026-11-01
- DeepSeek Details Always-On Agent Design With External Watchdog Session for Progress Monitoring — teortaxesTex · 2026-11-01
- YASAKANI inference engine claims 1.69–2.86x speed wins over SGLang, vLLM and llama.cpp on RTX 5090 — udmrzn · 2026-11-01
- MegaMoE kernel posts 4x per-GPU throughput vs baseline on 8xH200 DeepSeek V4.1 Flash — michellechen · 2026-11-01
- Starcloud hosts first factory tour of its new space datacenter facility — Scobleizer · 2026-11-01