A 397B MoE now runs on one RTX PRO 6000 with 96GB VRAM
mrstoatey · reddit · 2026-07-27
Krasis, a MoE-focused runtime for streaming very large models through limited VRAM, now runs Ornith-1.0-397B interactively on a single RTX PRO 6000 Blackwell 96GB.
What they measured
- 1,346 tok/s prefill at 10k tokens
- 2,354 tok/s prefill at 39,920 tokens
- 23.6 tok/s decode over 50 tokens
- 20.4 tok/s sustained decode over 250 tokens
- With Adaptive Cold Mass Pruning, decode improved to 25.7 tok/s while skipping only about 1.8% of routed probability mass on average
How it works
- Experts stay in CPU RAM and are dynamically brought into VRAM
- About 43% of routed experts were resident in VRAM in this run
- Peak process RAM was about 202GB, so the author says 256GB RAM is feasible on a consumer DDR5 motherboard
Additional notes
- Smaller MoEs that fit in system RAM can run much faster, e.g. 35B-class models at 117 tok/s decode on a 5090
- The same model can even run on a single RTX 5090 32GB at around 7.9 tok/s decode if you are patient
More from Infra
- fmgo: call Apple's on-device Foundation Models from Go with no CGO and no Swift — Super_Run_8466 · 2026-09-23
- Huawei unveils Peerium architecture: nested BSP unifies million processors into one computer — Dr_Singularity · 2026-09-23
- Grok explains why DeepSeek picked DualPipe + ZeRO-1 over ZeRO-3 on 2048 H800s — TheZachMueller · 2026-09-23
- AI costs fall 47% per quarter, 4x faster than DNA sequencing: Epoch AI — daveholtz · 2026-09-23
- M5 Ultra LLM test: 4x faster prompt processing, but double the power draw — DigitalguyCH · 2026-09-23
- $500 of Dell OptiPlexes become a diskless netboot lab where AI agents can't brick the hardware — colinmcnamara · 2026-09-23