Running an 816B MoE across 10 GPUs: pipeline parallelism beats tensor parallelism 4.6x on prefill
fuzhongkai · reddit · 2026-10-23
The author of TensorSharp, a .NET LLM inference engine, ran Muse Spark 816B (42B active) across 10 NVIDIA A40s and benchmarked parallelism strategies:
- Pipeline parallelism wins long-context prefill: 16K prompt prefill hit 1,346.8 tok/s vs 295.2 tok/s for tensor parallel (4.6x), while tensor parallel was 35.6% faster for single-request decode and 35.9% higher aggregate throughput at 4 concurrent requests.
- MoE amplifies communication costs: per-expert normalization means all-reduce must transfer individual expert outputs (8x more data), a major bottleneck over PCIe without P2P. Optimal strategy depends on the model's internal compute/communication patterns.
- Native whole-model execution via ggml's meta backend boosted decode from 1.9-2.4 tok/s to 22.63 tok/s (9-12x) — reducing execution overhead matters as much as kernel optimization.
- CUDA's 8-peer limit: wrapping cuMemSetAccess, keeping affected memory pools GPU-private and disabling P2P where needed let the model run on all 10 GPUs without patching upstream ggml.
Related event: TensorSharp Runs 816B MoE Model Across 10 A40 GPUs(2 posts)→
More from coding & agent
- Claude Code Pushes Urgent Security Update Across All Installations — umeshonai · 2026-10-23
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23