InferSSD streams MoE experts from SSD so you can run models bigger than your VRAM
AIFlow_ML · x · 2026-11-15
InferSSD, an open-source project built on llama.cpp, lets you run models larger than your RAM or VRAM by streaming MoE expert weights from an SSD on demand. It works with off-the-shelf GGUF weights and has been shown running Qwen3.6-35B and Qwen3.8 Flash Next.
- Supports macOS/Metal and Linux/CUDA, with Windows in progress
- Apache-2.0 licensed, 1.1k stars on GitHub
Because MoE models activate only a few experts per token, sequential SSD reads can suffice — offering a cheap path to local inference of large models on memory-constrained machines.
More from Infra
- Americans: no data centers in our backyard! SpaceX: no problem — draecomino · 2026-11-15
- The Starship math: 25,000 flights to deploy 250GW of space compute — JOBhakdi · 2026-11-15
- Jade: GPU-Accelerated Homology Search Engine, 26x Faster Than BLASTN — anshulkundaje · 2026-11-15
- Pre-training compute share collapses from 67% to 7% as inference and RL take over — robleclerc · 2026-11-15
- Nvidia's Huang told executives his Megaspeed interactions were standard business dealings — ShakeelHashim · 2026-11-15
- Three years of chip curbs: Huawei claims its AI chips now outsell Nvidia in China — Scobleizer · 2026-11-15