Edge0 Streams a 35B MoE from SSD at 20 tok/s on a Single 24GB GPU, Open Source
Edge0 · hf · 2026-09-17
Edge0 is a streaming MoE inference engine that tackles the memory wall of running large MoEs on consumer hardware: a 35B-class model needs 19.5GB at 4-bit, and naive SSD offloading fails because layer N+1's experts are chosen only after layer N's output exists.
- Prerouter: a per-layer head predicts next-layer routing one token ahead, and the prediction is consumed as the routing itself — the staged expert set exactly equals the routed set, nothing dropped
- An unmerged recovery LoRA trained on the student path pays back int4 quantization and routing-replacement quality loss
- Results: 20 tok/s on a single 24GB GPU within 3GiB peak active memory, within a few points of the fp16 teacher across five public benchmarks; an 8B tier runs on the same framework
- Framework, checkpoints, and adapters are open source
More from Infra
- The new AI stack starts below models: energy, grid and data centers — ingliguori · 2026-09-17
- US House Passes Bipartisan Bill Making Data Centers Cover More Grid Costs — Polymarket · 2026-09-17
- Dev trains a 2B Apache-licensed model on 24GB VRAM using OLMo embeddings and a 1B Engram table — NineThreeTilNow · 2026-09-17
- Applied Materials to invest $5bn in India as country boosts chip support — pstAsiatech · 2026-09-17
- Cloud AI Costs 10-20x More Than On-Premises, Expert Warns of Repeat of Early Cloud Mistakes — DavidLinthicum · 2026-09-17
- Huawei says AI chip demand outstrips supply, plans two new AI chips by 2027 — sunychoudhary · 2026-09-17