Running 300B MoE on 32GB RAM: Inference Bottlenecks and Optimizations
maddie-lovelace · reddit · 2026-08-09
The author shares optimization findings for running a 300B parameter MoE model on a 32GB RAM laptop. Due to memory limits, the massive expert pool (147GB) must stay RAM-resident.
Key findings and optimizations:
- Read speed is the primary bottleneck, not compute kernels. Repacking the model to a layer-major format converts slow random reads into large sequential reads at 7GB/s.
- Caching can counterintuitively slow things down: Page cache double-buffering transfers data from SSD to CPU then to GPU, worsening bandwidth usage by about 3×.
- Speculative expert prefetch at prefill: Using router hints to predict and prefetch the next layer's experts during the current layer's compute can effectively hide read latency.
- TTFT is inherently slow: Even a 30-token prompt touches a huge fraction of experts across 43 layers, meaning compute is completely masked by read times.
More from Infra
- Optimizing Local AI Video Models on RTX 3090: Best Configurations — BackgroundCow1411 · 2026-08-09
- LeCun Weighs In: Breakthrough AI Silicon Fails Without Software Ecosystem — ylecun · 2026-08-09
- Running LLMs on Snapdragon NPUs: A Guide to Qualcomm's GenieX CLI — carrycooldude · 2026-08-09
- AI Compute Costs: UK Datacentre Expansion Sparks Water and Power Crises — nordicinst · 2026-08-09
- Developer Creates MiniMax H3 RunPod Template for Easy Deployment — Draufgaenger · 2026-08-09
- Amazon's Planned Gas Plant for AI Data Center Could Become Top US Climate Polluter — Nunki08 · 2026-08-09