The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu, Xiaodong Zeng
cs.AI
2026-09-16
Edge0 predicts next-token expert routing, serving a 35B MoE at 20.4 tok/s in 2.9 GiB on a 24 GB Mac mini, 3.9 points behind its fp16 teacher.
On a desktop, MoE inference is a weight-memory problem. A 35B-class model is still 19.5 GB at 4-bit. Sparsity cuts FLOPs per token, not the bytes that have to live somewhere. Four bits is the usable floor; below that, post-hoc error wrecks the model.
Parking experts on SSD does not, by itself, hide disk latency. Layer N+1's expert set depends on layer N's output, so a naive stream stalls once per layer per token. On a Mac mini M4 Pro with 24 GB, a resident mlx-lm server holding the full 4-bit checkpoint decodes at 3.9 tok/s and occupies 18.2 GiB, leaving almost nothing for the OS and the KV cache.
Edge0 keeps expert weights on SSD and mmap-streams them into a bounded pool. Three design choices carry the system.
Prediction is the routing. A small head owned by layer N reads that layer's post-attention state at token t and predicts layer N+1's routing at token t+1. At decode, the MoE layer consumes that prediction and never runs the original router. The staged set equals the routed set, so nothing is dropped and no fallback load is required. The lead has to be a full token. Choosing the next layer right after attention, as Pre-gated MoE does, costs 30-100 ms of per-layer sync in this engine, more than the load it can hide.
Each head is fc1, erf-gelu, fc2, plus a linear residual warm-started from the next layer's router weights. The input concatenates the hidden state with this step's and the previous step's top-k one-hots. The 35B tier ships 33 heads (owners 6-38); 32 of 40 layers stream from a prediction.
Unmerged recovery LoRA. The int4 base stays frozen. A low-rank adapter is distilled from the fp16 teacher on the student path, paying for quantization and routing replacement together. Merge-then-requantize erases most of the adapter: 34% of the effect survives on an attention projection, 2% on a dense projection, 18% at the logits. Serving the delta in parallel costs 42 MB and no measurable decode time.
Training has three phases on the dequantized bf16 reconstruction of the served 4-bit checkpoint: distill the heads, SFT on about two million rows of teacher text with prerouting active, then on-policy distillation (reverse KL) on the student's own generations. Distillation-only checkpoints collapse into repetitive text.
The released 35B profile narrows the base model's K=8 to K=4 by swapping adapter files, without retraining the base.
Throughput is on a Mac mini M4 Pro 24 GB. Quality is OpenCompass against the fp16 bases under identical settings.
| System | Decode | Peak memory |
| Resident mlx-lm 4-bit | 3.9 tok/s | 18.2 GiB |
| Edge0 35B K=4 | 20.4 tok/s | 2.9 GiB |
| Edge0 8B | 28 tok/s | 1.5 GiB |
Disable the prerouter on that 24 GB box and decode is still 19.9 tok/s; the page cache already holds hot experts. The prerouter's real gain is on a 16 GB MacBook M2 where the 18.4 GiB checkpoint does not fit. Against pure on-demand streaming, K=2/4/8 move from 4.8, 3.5, 1.8 tok/s to 8.6, 6.4, 3.3 tok/s (+80%, +82%, +84%). At K=4, time blocked on expert loads falls from 244.0 ms to 101.9 ms per step.
| Benchmark | edge0-35b | Qwen3.6 fp16 | edge0-8b | Ling 3.0 tiny fp16 |
| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 |
| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 |
| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 |
| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 |
| IFBench | 57.9 | 61.7 | 53.9 | 60.6 |
| Average | 79.2 | 83.2 | 69.9 | 72.7 |
Mean per-benchmark gaps are 3.9 points (35B) and 2.8 (8B). MMLU-Pro on the 8B tier is 4.3 points above the teacher.
The usual answers for a 35B MoE on a desk are more RAM or a datacenter. Edge0 takes the other placement: experts live on SSD, a trained prerouter hides the reads under compute, and an unmerged LoRA pays the quality bill. Framework, checkpoints, and adapters are public.
It is a single-request engine on Apple Silicon. MLX is the only implementation; the CUDA slot is architecture, not code. Routing width K moves speed and memory together: the same cache budget at K=4 nearly doubles decode versus K=8 (6.4 vs 3.3 tok/s) on the 16 GB machine.
Do not expect an 80% prerouter bump on a machine whose page cache already holds the working set. On 24 GB the head barely moves throughput. It pays when the checkpoint does not fit.
One request at a time, FIFO. Batching changes the expert working set in ways these profiles do not model. Decode is floored by CPU-side graph building, 44 ms per step in the 40-layer forward; storage work cannot move that. Adjacent tokens agree on about a quarter of a layer's experts, so a prefetched expert is often read once and paid for twice.
Quality loss concentrates in long-chain reasoning: 6.1 AIME points on the 35B tier, 10.0 on the 8B. Every other 8B benchmark is within 6.7 points. Single-shot speed numbers on this hardware swing ±40% run to run and up to 2.3x across sessions, which is why every speed claim is a same-session rotated A/B median.