Offloading MoE models to RAM causes slow prefill speeds

former_farmer · reddit · 2026-08-24

Community discussions highlight that while offloading MoE models to system RAM allows decent decode speeds on GPUs with low VRAM, the prefill phase becomes extremely slow, significantly degrading the user experience. Users are confirming if this performance bottleneck is consistent.

Original post →

More from Infra

• Intel Expects Wins with MSFT, QCOM, MRVL on ASIC, CPU, CPO — BenBajarin · 2026-08-24
• Llama-Mobile: 2.7-Bit Quantization Shrinks Llama 3.2 Vision 11B to 3.7GB for Phones — Luka Ribar · 2026-08-24
• DSCO Router Launches Unified Gateway for Multi-Model Routing with BYOK Support — arthurcolle · 2026-08-24
• Open Source RobotSoul: Persistent Identity for Agents After Context Resets — robauto-dot-ai · 2026-08-24
• Etched Raises $1B Led by Jane Street to Validate Architecture-Agnostic AI Chips — TheTuringPost · 2026-08-24
• ConvRot Quant joins llama-cpp: Q6 accuracy nears Q8 quality — giveen · 2026-08-24

Infra channel →