Running an 816B MoE on 10× A40s: layer split beats tensor parallelism 4.6× on prefill
fuzhongkai · reddit · 2026-10-23
A developer ran Muse-Spark-Open-816B-A42B (316 GiB weights) across 10 NVIDIA A40s using TensorSharp, his .NET inference engine, benchmarking against llama.cpp:
- Layer split: 999.7 tok/s 4K prefill (2.07× llama.cpp), 1346.8 tok/s 16K prefill, 16.7 tok/s decode
- Tensor parallel: 22.6 tok/s decode and 45.7 tok/s with 4 concurrent requests, but only 295 tok/s prefill — a 4.6× prefill gap versus layer split
- The MoE architecture normalizes each expert's output before aggregation, so tensor parallelism must communicate individual expert outputs, exploding all-reduce traffic; on PCIe-only GPUs with broken P2P, bandwidth becomes the bottleneck
- Quality check: 98.8% top-1 agreement with llama.cpp over 12,276 teacher-forced WikiText-2 predictions
Caveat: llama.cpp's NCCL build failed on this machine, so its tensor-parallel numbers used the non-NCCL path — the comparison isn't fully apples-to-apples. Takeaway: tensor parallelism isn't automatically optimal for very large MoE models, especially on PCIe-only systems.
Related event: TensorSharp Runs 816B MoE Model Across 10 A40 GPUs(2 posts)→
More from Infra
- SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
- Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23
- IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23
- Jensen Huang on NeoCloud financing worries: high demand solves everything else — inductionheads · 2026-10-23