Benchmark: Swapping to MoE model boosts local agent throughput 8x
AIForOver50Plus · reddit · 2026-08-21
A benchmark on a MacBook Pro M3 Max comparing a dense 27B model against a 30B-A3B MoE model reveals massive concurrency gains for the MoE. While the dense model flatlines near 20 tok/s due to memory bandwidth constraints (reading all 27B params per token), the MoE model scales linearly with concurrent agents (reaching 158.6 tok/s with 8 agents) by activating only 3B params per token. The MoE also drastically reduces time-to-first-token (0.8s vs 32.2s), eliminating the "frozen" feeling during heavy loads.
More from Infra
- Haight Street Compute Market: a one-day trading floor for GPU deals in SF — ctjlewis · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- LMA brings day-0 model support to TPU for the first time, spanning NVIDIA, AMD, Intel via SGLang — ying11231 · 2026-10-16
- Semtech's analyst-day slide cleanly explains the four signal-integrity impairments — BenBajarin · 2026-10-16
- Fortytwo Engine boosts Qwen3.8-27B accuracy 7% and cuts tokens 25% via Swarm Inference — rohanpaul_ai · 2026-10-16
- Berkeley scholars: concentrated chip supply chain makes a hardwired frontier AI pause feasible — wfithian · 2026-10-16