TokenWing offloads 60% of MoE experts to SSD, 120B at 50 tok/s on a laptop
Grawnch · reddit · 2026-11-19
TokenWing, an open-source inference engine with a GUI wrapper called Run My AI, streams over 60% of routed MoE experts from SSD with low latency. User-reported benchmarks, all without MTP:
- Ling 3.0 Flash 120B MXFP4: 45-50 tok/s decode, 900 tok/s prefill on 12GB VRAM + 32GB DDR5
- Qwen 3.8 Flash Next (3.5bpw): 35-40 tok/s decode on 8GB VRAM + 32GB DDR5
- Flint 40B UD-Q4KXL: 30 tok/s decode on 24GB DDR4 + 4GB VRAM
The project is early stage — the autotuner is still WIP, so manual flag tuning is recommended for now. Since the engine only reads (never writes), SSD wear is not a concern.
More from Infra
- Embedl cuts Jetson Thor driving-plan latency from 603.7ms to 82.7ms without touching weights — AIFlow_ML · 2026-11-19
- NVIDIA cuda-tile bug: LoopSplit arithmetic wraps at -O3, silently miscompiling GPU loops — vxnuaj · 2026-11-19
- Compute goes financial: $5M B200 options, Akash Compute Forwards, $45M raise — 0xJeff · 2026-11-19
- Fed's Barr flags AI circular financing: top 7 AI firms' debt to top $800B — SumitGup · 2026-11-19
- Jensen Huang pushes back on WSJ's 'lack of substance' claim over $500B pledges — basedjensen · 2026-11-19
- Japan tech equipment orders surge unusually smoothly on global AI demand — PAstynome · 2026-11-19