Running 300B MoE on 32GB RAM: Inference Bottlenecks and Optimizations
maddie-lovelace · reddit · 2026-08-09
The author shares optimization findings for running a 300B parameter MoE model on a 32GB RAM laptop. Due to memory limits, the massive expert pool (147GB) must stay RAM-resident.
Key findings and optimizations:
- Read speed is the primary bottleneck, not compute kernels. Repacking the model to a layer-major format converts slow random reads into large sequential reads at 7GB/s.
- Caching can counterintuitively slow things down: Page cache double-buffering transfers data from SSD to CPU then to GPU, worsening bandwidth usage by about 3×.
- Speculative expert prefetch at prefill: Using router hints to predict and prefetch the next layer's experts during the current layer's compute can effectively hide read latency.
- TTFT is inherently slow: Even a 30-token prompt touches a huge fraction of experts across 43 layers, meaning compute is completely masked by read times.
More from Infra
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- Neocloud M&A decoded: CoreWeave, Nebius, Crusoe, Nscale all bet margins vanish beyond GPU scarcity — AccBalanced · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- WSL 3.0 ships with native containers and Linux 6.18 kernel, syscall overhead down ~16% — solyarisoftware · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16