DeepSeek inference internals: decode stays on wideEP ep64, prefill routes without cache awareness

teortaxesTex · x · 2026-10-20

Engineer SuJinYan123 reviews DeepSeek's newly open-sourced inference-side design, drawing on the repo and a detailed Zhihu post:

• Decode still runs classic wideEP (ep64); load balancing is roughly a kv cache + num running combination, tuned case by case.
• Prefill load balancers are the highlight: production-style bucketing (previously explored by Together), routing with no cache-awareness (pooling as fallback), and dp groups balancing directly on compute tokens.
• A notable time-estimator idea: fix "duration" instead of chunk size, run an offline profiling pass, then let the scheduler estimate and batch requests.
• The author argues PP suits small/mid deployments for prefill while EP only pays off at large scale; accompanying notes (images) cover agentic workload long-tail distributions, long/short request pool splitting, cache-hit selection, and mooncake store incast handling.

Original post →

More from Infra

• Latency test from 6 cities: Jev beats OpenAI Decisions and Cloudflare Clef — ImaginaryRoyal9725 · 2026-10-20
• Raw data shows OpenAI returned 271 overload errors to a Codex pool in 48 hours — rickasaurus · 2026-10-20
• Researcher claims 4x faster convergence to backprop loss using Flywheel infra — tensorqt · 2026-10-20
• Chrome 155 ships JPEG XL: 30-50% smaller than JPEG with lossless JPEG transcode — banteg · 2026-10-20
• Polymarket: DeepSeek releases open-source tool to sharply cut AI inference compute costs — Polymarket · 2026-10-20
• Together, Fireworks, Prime Intellect and Modal are all converging on inference + RL training — NielsRogge · 2026-10-20

Infra channel →