Redditor sketches a looped small MoE with SSD-resident engram tables and Dorry KV optimization
Sea-Wind-2216 · reddit · 2026-11-18
A Redditor proposes a speculative small MoE architecture (10B total, 2B active) combining three existing ideas:
- MoE sparsity per loop for better scaling than dense models
- Engram tables: 20-50B of static knowledge parameters that can live on SSD, adding cheap knowledge while keeping active params low
- Looped inference to boost reasoning
- NVIDIA's Dorry-style KV cache optimization to fix the KV growth that looping causes
The goal is to hide looping costs with tiny active params so it runs on low-VRAM cards; the author hopes a future Qwen4 MoE adopts some of this. Informal brainstorm, no experiments.
More from Infra
- TokScale leaderboard: #1 user burned 4.65T tokens, est. $2.43M in cost — olcan · 2026-11-18
- Inside Microsoft's massive Outlook outage: what broke and why fixes took so long — tomwarren · 2026-11-18
- vLLM ships DSpark draft model for Kimi K3.1 Pro with up to 30% higher acceptance length — vllm_project · 2026-11-18
- Wonderful runs in-house voice models on Together AI at 274ms first sentence — togethercompute · 2026-11-18
- Anthropic's S-1 Shows Only 25% of Compute Goes to Inference — Sparking a Bear Case — toptickcrypto · 2026-11-18
- LanceDB Raises $57.5M Series B for AI-Native Multimodal Data Stack — pzakin · 2026-11-18