Tuning Kimi K3 on H200s: TP+EP Beats DP+EP for Throughput
vwxyzjn · x · 2026-08-01
The LMSYS team shared engineering insights on tuning the Kimi K3 model for throughput-oriented workloads on H200 clusters:
- TP+EP beats DP+EP: Since only about 1/4 of K3's layers use gated MLA, DP attention provides less benefit from avoiding KV-cache duplication and increases latency. This differs from their preference for DP+EP in K2.6 and GLM setups.
- FP8 KV cache worked well: They measured no quality or throughput regression in evaluations while effectively doubling KV-cache capacity.
- TP32+EP32: Provided significant performance improvements (details in the original post).
Related event: Kimi K3 Throughput Optimization on H200 Clusters(3 posts)→
More from Infra
- Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0 — zhyncs42 · 2026-10-16
- Nemotron 3.5 Bolt runs 16 concurrent streams on a single DGX Spark — NVIDIAAI · 2026-10-16
- Why your computer can't store 0.2: an IEEE 754 primer — Franc0Fernand0 · 2026-10-16
- Sabaki: MIT-licensed typed decision router runs on CPU in milliseconds — Fox3000foxy · 2026-10-16
- BitRun v0.2 runs 8B ternary LLMs on CPU in 2GB RAM with live wattage telemetry — The_VibePreneur · 2026-10-16
- MiniCPM-V 4.5 MoE reportedly hits 13+ tokens/s on iPhone with just 3GB memory — heyshrutimishra · 2026-10-16