pMLX Optimizes Qwen and GLM with Expert Paging, Runs Large MoEs on 37GB RAM
EyalToledano · x · 2026-08-29
The article covers updates to the pMLX project, optimizing inference for large OSS models (180B-300B class) like Qwen 3.8 Flash and GLM-5 on Apple Silicon.
Key Technical Breakthroughs:
- Expert Paging: A (new?) technique where experts stream from NVMe into memory on demand rather than residing in RAM full-time, applying n-gram streaming concepts to MoE architectures.
- Memory Efficiency: Applied to the full model without pruning, 60% of experts live on the SSD, allowing the model to run in 37GB of RAM.
- MTP Performance: pMLX makes MTP (speculative decoding) profitable on M4 chips. Checking the drafter's guesses now costs barely more than a single step, doubling decode speeds compared to mlxvlm.
Releases:
- REAP-288-bf16 + GGUF, MXFP4, and NVFP4 quantizations are now available on Hugging Face.
More from Infra
- Analyzing NVDA Valuation: AI Spend Sustainability and Margin Compression — menhguin · 2026-08-29
- Samsung Unveils LPDDR5X-PIM: 614GB/s In-Memory Bandwidth — jedisct1 · 2026-08-29
- Deploy Models with Kubernetes and TensorFlow Serving — Al_Grigor · 2026-08-29
- Serverless Deep Learning: Deploy Models on AWS Lambda — Al_Grigor · 2026-08-29
- Model Deployment: FastAPI, Docker, and Cloud Deployment — Al_Grigor · 2026-08-29
- Firefox & Chrome intend to ship support for JPEG XL — addyosmani · 2026-08-29