Dynamic REAP swaps 4 experts every 256 tokens, keeping only 25% in GPU memory
Tim_Dettmers · x · 2026-09-22
Compressed models deploy with Dynamic REAP: even keeping only 25% of experts per layer, routing probabilities of removed experts faithfully indicate which experts the current distribution needs, guiding async transfers from CPU RAM/NVMe to GPU. Four experts are swapped every 256 tokens at token boundaries without disturbing inference. Surprisingly, this yields better out-of-distribution generalization than fitting REAP on the test set — MoE routing distributions change frequently as documents are processed (agent work alternates docs, code, user interaction).
Related event: Dettmers Explains Iterative Sensitivity Probing and Dynamic REAP(2 posts)→
More from Infra
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12
- Megakernel is multiprogramming for GPUs: 'GPUs are entering the CPU age' — pranjalssh · 2026-11-12