Tideway: phase-aware unified memory pool speeds MoE inference on consumer devices by ~2x
DukeUniversity · hf · 2026-11-19
A Duke University-led team presents Tideway, a system for efficient inference of large Mixture-of-Experts models on memory-constrained consumer devices.
- Context: Deploying large MoE models locally balances privacy, API cost, and efficiency, but consumer devices are tight on both VRAM and host RAM, forcing expert weights to spill across VRAM, RAM, and SSD. Existing systems treat memory tiers as independent caches and thrash during inference.
- Core idea: Manage host RAM and GPU VRAM as a unified memory pool, coupling phase-aware memory layouts that adapt to distinct prefill/decode access patterns with optimized CUDA Graph execution under expert paging.
- Results: On representative consumer hardware with frontier MoE models, Tideway averages 1.97× end-to-end latency, 2.40× TTFT, and 1.61× TPOT speedups over the fastest baseline. Source code is publicly available.
More from Infra
- AMD ROCm 10.2 Ships With 18 New Ryzen Laptop Models Supported for Local AI — AnushElangovan · 2026-11-19
- Consumer NAND spot prices drop 20% in a month, more SSD relief expected in early 2027 — SumitGup · 2026-11-19
- Report: Nvidia sales staff suggested Alibaba rent A100s via Megaspeed, now under criminal probe — basedjensen · 2026-11-19
- CriticNPU: Mechanistic critic halves simulator calls for sample-efficient NPU design-space exploration — Yicheng He · 2026-11-19
- Qwen3.8-27B opencode-adapted ported to MLX: up to 73 tok/s on M5 Max — yb2698 · 2026-11-19
- Ex-Siri platform engineer: one system runs Siri across device and server — jdluk87 · 2026-11-19