Specter speeds single-GPU MoE inference with quantized multi-step speculative offloading
Wenfeng Wang · hf · 2026-10-21
A Shanghai Jiao Tong University team presents Specter, an inference system for MoE models that exceed single-accelerator memory. Since data-dependent routing keeps expert transfers on the critical path, offloading to host memory creates severe PCIe I/O bottlenecks.
Specter combines quantization-enabled speculative decoding with multi-step speculative expert prefetching to hide PCIe latency and improve single-GPU MoE throughput.
More from Infra
- Morgan Stanley: hyperscaler AI capex to hit $3.4T across 2027-2028 with leases included — FinanceYF5 · 2026-10-21
- MiniMax's M3.1 Flash talk: a faster model isn't faster if the job isn't — AI Engineer · 2026-10-21
- Training Factory open-sources QLoRA fine-tuning on 4-bit CQ-quantized models — textclf · 2026-10-21
- DeepSeek open-sources DeepInfer: 155T tokens/day, $38/GPU-hour, >80% margin — Latent Space · 2026-10-21
- Cerebras WSE-3 Turbo packs 4T transistors and 250 PFLOPS into a wafer-scale chip — GregoryDiamos · 2026-10-21
- Anthropic reportedly built Scion to replace Git for 30,000 daily agent merges — Capatain_CV · 2026-10-21