LayerStoRm open-source engine runs 186GiB MoE on 96GB VRAM at 24.5 tok/s with 1M context
CharacterBumblebee99 · reddit · 2026-09-07
A Reddit user released LayerStoRm, an experimental MIT-licensed inference engine that streams MoE experts from pinned host RAM to GPU per token, letting you run models far larger than your VRAM.
- Demo: GLM-5.3-Flash UD-Q4KXL (186 GiB of weights) on 2×RTX 5090 + 2×RTX 5080 (96 GB VRAM total, 208 GB pinned RAM) at 1M context — 24.5 tok/s decode @8k, 159 tok/s prefill @27k
- CPU does no compute, only feeds experts; transfers are NUMA-aware and eventually PCIe-bound
- Built for agentic coding: prefix caching with mid-prompt checkpoints cuts re-prefill TTFT from 67.5s to 18.4s @8k, and 923s to 79s @97k
- One-command autoconfig derives the whole recipe from weights + hardware; currently NVIDIA SM120 only
Repo: github.com/kkontosis/LayerStoRm
More from coding & agent
- WebMCP explained: giving in-browser agents a labeled front door instead of screenshot-clicking — thisiskp_ · 2026-09-07
- Inside x.ai's Haggle Bot: how an agent found $100K+ in procurement savings — FinanceYF5 · 2026-09-07
- Grok Bot opens template marketplace; Haggle Bot found $100K+ in savings in a week — FinanceYF5 · 2026-09-07
- Grok Bot opens template marketplace; Haggle Bot found $100K+ in savings in a week — FinanceYF5 · 2026-09-07
- Developer builds AI agent harness on iMessage that trades and pays via PayBox — kleffew94 · 2026-09-07
- Codex tasked with designing its own sheet-metal part via DFM/quote MCP — PaulYacoubian · 2026-09-07