Redditor sketches a looped small MoE with SSD-resident engram tables and Dorry KV optimization

Sea-Wind-2216 · reddit · 2026-11-18

A Redditor proposes a speculative small MoE architecture (10B total, 2B active) combining three existing ideas:

The goal is to hide looping costs with tiny active params so it runs on low-VRAM cards; the author hopes a future Qwen4 MoE adopts some of this. Informal brainstorm, no experiments.

Original post →

More from Infra

Infra channel →