antirez: prefetching MoE experts via previous-layer router lets K3.1 stream on a 128GB box
antirez · x · 2026-11-05
antirez describes running K3.1 in streaming mode on a very small (128GB) machine: by running the next layer's router with the previous layer's residual for expert prefetching, the top-4 experts are almost always predicted correctly by score, enabling SSD read hints and efficient inference under tight memory.
More from Infra
- New AI chip startup emerges from team of ex-Google TPU engineers — nmasc_ · 2026-11-05
- Claim: CoreWeave Cut Off From Bond Markets, Unable to Meet 2027 Obligations — SumitGup · 2026-11-05
- Bratton: "Against Data Centers" Is the "No Nukes" of the 2020s — and Will Backfire — bratton · 2026-11-05
- Cornell sets semiconductor bandgap record with 7+ eV doped aluminum gallium oxide devices — AviatorAshley · 2026-11-05
- SiTime revenue hits $298.7M, up 257% YoY — BenBajarin · 2026-11-05
- George Hotz to teach CMU course where students build an ML compiler from scratch up to training LLMs on GPUs — QuixiAI · 2026-11-05