Prime Mega MoE extends DeepSeek's kernel for training, up to 34% faster end-to-end MoE training
cfregly · x · 2026-11-02
Prime Intellect released Prime Mega MoE, extending DeepSeek's fused Mega MoE kernel (from the open-source DeepGEMM collection) with a fast overlapping backward pass to make it usable for training: up to 3.11× faster MoE kernels vs the default PyTorch implementation and up to 1.34× faster end-to-end training vs the Prime-RL baseline (34% higher throughput).
Key details
- In MoE training, tokens are routed across GPUs to their experts twice per layer (forward and backward). Normally sorting, transport, expert math, and return are separate serialized operations, wasting GPU time waiting.
- Mega MoE replaces this chain with a single kernel invocation that routes tokens, runs expert MLP math, and combines results in one pass — no GPU↔CPU round trips, overlapping compute with communication.
- On Qwen3-30B-A3B with 32 H200s, step time fell from 2.78s to 2.08s (25% shorter); baseline used torch all-to-all + DeepGEMM FP8, both paths with fullgraph compilation.
cfregly notes the kernel is closed-source and that communication and expert math share GPU capacity — he advises starting from Prime's defaults and comparing full training steps for correctness.
More from Infra
- OpenAI may rebrand its data centers as 'AI compute campuses' — TechCrunch AI · 2026-11-02
- $300B Louisiana project pipeline squeezes steel supply, H-beam lead times hit 9 months — MatthewChang · 2026-11-02
- Benchmark: Speedy inference engine crushes Avalon — 1.6x faster reads on Qwen3 long context — MotorNetwork380 · 2026-11-02
- vLLM serves DeepSeek-V4.1-Flash at 954K tokens/s per GPU on one GB300 NVL72 rack — bookwormengr · 2026-11-02
- 2.3bpw quant update: 27B model on 12GB cards ties Qwen3.6 BF16 on LiveCode Bench at 84.3% — Brief-Tap-6616 · 2026-11-02
- AI datacenters priced at ~$50B per gigawatt, but build speed is the metric that matters — JOBhakdi · 2026-11-02