177B MoE on a 16GB GPU: SSD streaming hits 9-10 tok/s with custom inference engine
TypicalPudding6190 · reddit · 2026-09-28
The Inferred-Thoughts team built an inference engine that runs MoE models too big for VRAM+RAM by streaming experts from SSD. On a RTX 5060 Ti 16GB + 32GB RAM + Gen5 NVMe, they run Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s decode — double llama.cpp's 4.9 tok/s on the same machine.
Memory layout: dense weights (4.4 GiB) and hottest experts (8.8 GiB) in VRAM; embeddings and next-tier experts in pinned RAM; 48.5 GiB of routed experts plus a 50.7 GiB n-gram table stay on SSD, read on demand.
Key techniques and numbers:
- GCLOCK eviction, lookahead prefetch of next-layer experts
- FP4×FP4 matmuls on tensor cores without unpacking
- Each token uses 480 experts; 377 hit VRAM/RAM, 103 (270 MiB) come from SSD — a 75% hit rate that makes 9 tok/s possible
- 49.2 tok/s prefill; near-zero SSD writes, though the drive hit 70°C on long runs
- Qwen3.6-35B-A3B NVFP4 also supported at 47.3 tok/s decode with tuned config
Limits: RTX 50-series (sm120) only, Windows 11/WSL2 only, greedy decoding. Ships with an OpenAI-compatible server with tool calls and a built-in chat UI. The team expects v2 to reach 14-15 tok/s. Repo and weights are public.
More from Infra
- Sarvam AI Launches Token Factory: India-Hosted Inference for Open Models — itsOmSarraf_ · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16