Threadripper PRO benchmark: 24 Zen 5 cores saturate 8-channel DDR5 for CPU MoE offload

mr_il · reddit · 2026-09-10

Planning a PC to run MoE models far beyond VRAM, the author rented a Threadripper PRO 9975WX (32-core Zen 5, 8-channel DDR5) on vast.ai and benchmarked FreeToken's production W4A8 dsfp4 GEMV kernel against synthetic expert banks matching DeepSeek V4.1 Flash's geometry (384 routed experts × 18.8MB ≈ 6.7GiB, 6 experts/token).

Key findings

Cores stop mattering at 24: 32 vs 24 cores yields +2.2–2.4% burst, 0% sustained; the workload saturates memory, not cores. The $1,200 cheaper 9965WX is the rational pick.
The FP4 kernel reaches 80% of STREAM bandwidth (178–182 GB/s vs 222 GB/s at 24 cores).
Thermal throttling is real: after 8 minutes of sustained load, throughput drops 17% (182 → 150 GB/s) with clocks unchanged — DRAM/IMC thermal throttling. DIMM airflow may literally be worth 20% decode speed.

In tokens/s (4.5GB routed experts per token, 24 cores): 33 tok/s at 0% expert cache hit, 83 at 60%, 133 at 75%. Versus streaming misses over PCIe 5.0 x16 (50 GB/s), the CPU path is 3–3.7× faster per missed expert — which is why FreeToken's hybrid mode computes most misses on CPU.

TL;DR: buy memory channels and DIMM cooling, not cores; the 32-core upgrade is +2% at best. Methodology and raw CSVs are published; caveats include rented single machine and container environment.

Original post →

Threadripper PRO benchmark: 24 Zen 5 cores saturate 8-channel DDR5 for CPU MoE offload

More from Embodied