LayerStoRm open-source engine runs 186GiB MoE on 96GB VRAM at 24.5 tok/s with 1M context
CharacterBumblebee99 · reddit · 2026-09-07
A Reddit user released LayerStoRm, an experimental MIT-licensed inference engine that streams MoE experts from pinned host RAM to GPU per token, letting you run models far larger than your VRAM.
- Demo: GLM-5.3-Flash UD-Q4KXL (186 GiB of weights) on 2×RTX 5090 + 2×RTX 5080 (96 GB VRAM total, 208 GB pinned RAM) at 1M context — 24.5 tok/s decode @8k, 159 tok/s prefill @27k
- CPU does no compute, only feeds experts; transfers are NUMA-aware and eventually PCIe-bound
- Built for agentic coding: prefix caching with mid-prompt checkpoints cuts re-prefill TTFT from 67.5s to 18.4s @8k, and 923s to 79s @97k
- One-command autoconfig derives the whole recipe from weights + hardware; currently NVIDIA SM120 only
Repo: github.com/kkontosis/LayerStoRm
More from coding & agent
- Developer Moves All Prompts to Skills: AI Coding Workflow v3 Explained — rseroter · 2026-10-26
- charlie Marsh: rewrites beat external deps in the AI era — mohamedmansour · 2026-10-26
- OpenAI and Twitch launch Codex hackathon with $25,000 in prizes — OpenAIDevs · 2026-10-26
- Filtering secrets on the write path: why agent memory deserves more suspicion at ingest than retrieval — Classic-Being-7960 · 2026-10-26
- INT21's AI agent swarm rewrote Drama 3's inference runtime in Rust within one hour, beating SGLang-Omni — bingxu_ · 2026-10-26
- After OpenCode key leak: 5 practical rules for keeping keys away from agents — jacobholm_dev · 2026-10-26