177B MoE on a 16GB GPU: SSD streaming hits 9-10 tok/s with custom inference engine
TypicalPudding6190 · reddit · 2026-09-28
The Inferred-Thoughts team built an inference engine that runs MoE models too big for VRAM+RAM by streaming experts from SSD. On a RTX 5060 Ti 16GB + 32GB RAM + Gen5 NVMe, they run Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s decode — double llama.cpp's 4.9 tok/s on the same machine.
Memory layout: dense weights (4.4 GiB) and hottest experts (8.8 GiB) in VRAM; embeddings and next-tier experts in pinned RAM; 48.5 GiB of routed experts plus a 50.7 GiB n-gram table stay on SSD, read on demand.
Key techniques and numbers:
- GCLOCK eviction, lookahead prefetch of next-layer experts
- FP4×FP4 matmuls on tensor cores without unpacking
- Each token uses 480 experts; 377 hit VRAM/RAM, 103 (270 MiB) come from SSD — a 75% hit rate that makes 9 tok/s possible
- 49.2 tok/s prefill; near-zero SSD writes, though the drive hit 70°C on long runs
- Qwen3.6-35B-A3B NVFP4 also supported at 47.3 tok/s decode with tuned config
Limits: RTX 50-series (sm120) only, Windows 11/WSL2 only, greedy decoding. Ships with an OpenAI-compatible server with tool calls and a built-in chat UI. The team expects v2 to reach 14-15 tok/s. Repo and weights are public.
More from Infra
- Spectral deflation framework improves Muon: consistent validation loss gains in GPT-2 pretraining — hankyang94 · 2026-09-28
- First Audited Look at Inference-Economics: MiniMax Hit 24.6% Margin, Peer Lost 75% of OpenRouter Volume — AccBalanced · 2026-09-28
- Yunnan Germanium report: indium for InP is tight in China, export controls aren't the bottleneck — pstAsiatech · 2026-09-28
- Apple's free on-device fm paired with decision model Jev beats big-model routing in tests — jasonkneen · 2026-09-28
- Estimate: crudely describing human biology needs 1000x more data than humanity stores — IgorCarron · 2026-09-28
- Cooling setup runs 6x RTX 6000 at full 325W for 3 months, GPUs at just 41°C — TheZachMueller · 2026-09-28