A $2.5k RTX 3090 box runs 328GB GLM-5.3-Flash locally via expert offloading
casper_hansen_ · x · 2026-11-05
casperhansen shares a $2.5k local inference build (used RTX 3090, 64GB DDR5, used SN850X, Ryzen 5 7600X) claiming 150 tok/s for Qwen3.8 27B, 15 tok/s for DeepSeek V4.1 Flash, and 10 tok/s for GLM 5.3 Flash — 'infinite private tokens' at home.
The companion open-source project Big-MoE-Small-Box shrinks 328GB GLM-5.3-Flash to a 125GB 3-bit quant and serves it from one 24GB GPU + 64GB RAM + NVMe at 7-10 tok/s with up to 512k context. Experts are tiered across free VRAM cache, pinned RAM, and an ODIRECT-backed NVMe store; a scheduler decides per layer whether each of the 8-of-288 routed experts runs from VRAM, is copied in from RAM/NVMe, or executes on CPU with AVX2. Built on 0xSero's glm-flash-lite. Current bottleneck is SSD read speed; a PCIe 5.0 SSD should push it to 25 tok/s.
Related event: $2,500 Local Rig Runs 27B Model at 150 tokens/s(2 posts)→
More from coding & agent
- Levels decompiles Battlefield 1942 with Claude Code, ports it to browser with working multiplayer — huangyun_122 · 2026-11-05
- Kubernetes promotes AI Conformance to permanent SIG Architecture subproject — TerryTangYuan · 2026-11-05
- quiet.md goes open source: Rust+GPUI editor, Feishu-compatible, ~5ms frames on 1M-char docs — dotey · 2026-11-05
- A background agent that auto-structures messy scientist Excel data, day 17 of agent series — nlarusstone · 2026-11-05
- Code mode isn't about MCP: it's about whether tool results go to context or program variables — ritakozlov · 2026-11-05
- Agent Arena: frontier models show 55-minute gap on long-horizon agent tasks — arena · 2026-11-05