Big-MoE-Small-Box: serve 361B DeepSeek and 321B GLM from one 24GB GPU via NVMe experts
casper_hansen_ · x · 2026-11-08
casperhansen open-sourced Big-MoE-Small-Box (Apache-2.0/MIT), serving MoE models far larger than the GPU from one 24GB card, 64GB RAM and an NVMe drive via an OpenAI-compatible server on exllamav3 1.6.6: GLM-5.3-Flash (321B, EXL3 3.05 bpw, 125GB) and DeepSeek-V4.1-Flash (EXL3 2.25 bpw, 361GB incl. 203GB Engram tables). Key idea: each token activates only a few routed experts (8/288 for GLM, 6/384 for DeepSeek), so the routed experts — most of the weights — are packed per layer into fixed-size records read with ODIRECT across a VRAM/RAM/NVMe tier store. Build list: used RTX 3090, 64GB DDR5-5600, WD SN850X, Ryzen 5 7600X. He invites anyone to copy the setup for "sovereign AI at home."
More from coding & agent
- Dev workflow tip: route CUA and search-heavy tasks to sol via subagents — corbtt · 2026-11-08
- Dimillian points to OpenAI Showcase for prompts, plans original demo-and-prompt series — Dimillian · 2026-11-08
- Testing ICE Constrained Decoding: Enforce Constraints at Generation, Not After — manvigupta03 · 2026-11-08
- ResearchBoost Agent speeds up real research workloads 2.51x on average across 12 tasks — YouJiacheng · 2026-11-08
- Built an interactive 3D optics course for GCSE physics entirely with Claude Opus 5.5 — techartist_ · 2026-11-08
- Perplexity CEO uses an agent to unmask anonymous recruiter candidates and reach them directly — AravSrinivas · 2026-11-08