Heterogeneous Deployment Strategy for MoE Inference

techNmak · x · 2026-07-12

Core Approach

• In MoE models, only a small subset of experts is activated per token (e.g., 8 out of 256 experts are activated in the article's example).
• Therefore, it's not cost-effective to fit all experts into expensive GPU memory.
• KTransformers' approach:
• GPU handles attention and shared components
• CPU memory hosts the large expert pool
• Frequently used experts can be placed back on GPU, rarely used ones on CPU
• Uses AMX/AVX to accelerate CPU-side INT4/INT8 inference
• NUMA-aware scheduling to reduce memory access overhead

Further Practice

• The project supports integration with SGLang for serving.
• Also supports integration with LLaMA-Factory for fine-tuning large MoE models under limited GPU memory.
• Examples given:
• Fine-tuning DeepSeek-V3/R1 requires about 4 RTX 4090 cards, total 80GB GPU memory
• Qwen3-30B-A3B can run on a single RTX 4090
• The project claims 17K+ GitHub stars and supports models like GLM-5.2, MiniMax-M3, DeepSeek-V4-Flash, Kimi-K2.5.

Conclusion

The author emphasizes: GPU memory should not hold the entire model, but the "working set". For MoE, a reasonable CPU-GPU heterogeneous division is more critical than simply piling on GPUs.

Original post →

More from Infra

• Tabul AI launches Metal TreeSHAP to speed up Shapley values on Apple silicon — Scobleizer · 2026-07-22
• DeepSeek-V4-Flash tops out at 770 tok/s on one B300 in a vLLM batch test — Moreh · 2026-07-22
• NVIDIA starts shipping 102.4 Tbps Spectrum-6 switches for Vera Rubin AI factories — nvidia · 2026-07-22
• Apple publishes SOC 3 audit reports for Private Cloud Compute — throwfaraway4 · 2026-07-22
• Reddit GPU renters say existing platforms only give you two of three: code, recovery, fair billing — legendpizzasenpai · 2026-07-22
• The Sandboxing Manifesto: Secure Execution Environments for Agents — spirosoik · 2026-07-22

Infra channel →