OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
Liujianfu Wang, Yuyang Du, Yuchen Pan, Soung Chang Liew, Jiacheng Liu, Kexin Chen
cs.DC
2025-12-04
OD-MoE's shadow model predicts Mixtral routing at 99.94% recall, loading experts just in time at 3.69 tok/s (75% of full GPU) with under 1 GB per worker.
Edge MoE is a memory problem. Expert offload keeps most parameters in CPU DRAM and caches a hot subset on the GPU. The reserved cache is often a worse use of VRAM than a dense model, and a wrong prefetch stalls the pipeline for a reload. EdgeMoE, HOBBIT, and Mixtral-Offloading shrink the copy with quantization; AdapMoE skips uncached experts. Quality takes the hit.
OD-MoE refuses a long-lived expert cache. Load an expert just before it runs, evict it immediately after. That only works if routing predictions are extremely accurate and if load and compute can be staggered across machines.
Three node roles. The main node holds attention, gates, and norms. A shadow node runs a quantized Mixtral as a faster emulator. Workers fetch predicted experts into GPU memory and run them. The testbed is one main RTX 3090, one shadow with two 3090s, eight worker 3090s, and 1 Gbps Ethernet. Mixtral-8×7B is top-2, so workers form four groups of two. While one group computes layer ℓ, the other three load ℓ+1, ℓ+2, and ℓ+3.
Scaled Emulative Prediction (SEP) treats the shadow's already-unfolded future routing as lookahead for the full-precision model. Quantization drift accumulates in autoregression, so the shadow's tokens and KV cache are periodically aligned to the main model. Aligning every step yields 99.94% recall with an FP16 shadow, 97.34% INT8, 95.67% NF4. Without alignment, recall falls to about 30% by token 256. Alignment delays the shadow's start, so the first layers have no prediction and wait on I/O. On 3090 workers, align-every-step is fastest; with 3080 workers, KV period 4 and token period 1 wins. Prior predictors: AdapMoE 86%, DAOP 84%, HOBBIT 91% averaged over up to four layers.
Prefill does not predict. A 16-token prompt activates 7.6 of 8 experts on average; 128 tokens activate all eight with 99.8% probability. Each of eight workers holds one expert per layer, and large batches are split into mini-batches so compute overlaps embedding transfer on the LAN.
Baselines were reproduced on an eight-3090 server (offload systems use one GPU). Decode is the headline metric, averaged over four (input, output) length pairs.
| System | Decode tok/s | End-to-end tok/s | GPU memory |
| Transformers, fully cached | 4.8900 | 4.8425 | 180 GB |
| OD-MoE | 3.6925 (75.51%) | 3.3700 | 60 GB |
| AdapMoE | 3.1300 | 3.0350 | 8 GB |
| Mixtral-Offloading | 2.2375 | 2.1725 | 11 GB |
| llama.cpp | 0.8225 | 0.7975 | 0 (CPU) |
Versus Mixtral-Offloading, MoE-Infinity (0.6875), HOBBIT (0.7850), and AdapMoE, decode is 1.65×, 5.37×, 4.70×, and 1.18×. Mean TTFT is 2244 ms, slower than Mixtral-Offloading at 1845 ms and AdapMoE at 1387 ms; those two win first token by quantizing copies, and they lose quality. AdapMoE MMLU 48.60%, GSM8k 22%; HOBBIT GSM8k 35%. OD-MoE uses the same full-precision weights as Transformers: MMLU 70.34%, Hellaswag 76.25%, GSM8k 64.14%. The 60 GB splits as 7 GB main, 45 GB shadow, about 1 GB × 8 workers.
Hot-expert caches spend VRAM on a guess. Once routing recall is above 99%, the cache can go away and a worker with under 1 GB can join the pipeline. That is a recipe for a rack of cheap GPUs, not a single laptop. The shadow's two 3090s and 45 GB are a real cost; do not quote only the per-worker number.
The testbed is 1 Gbps wired Ethernet, not Wi-Fi or cellular, so the delay model does not transfer. The shadow's 45 GB makes the "edge" story generous: workers are tiny, the system is not. Prefill TTFT lags quantized offload. Alignment period depends on compute versus PCIe and must be reswept per GPU. The GitHub link is a double-blind placeholder. No router or camera ran Mixtral; sub-1 GB IoT is an extrapolation.