Benchmarks show prefill/decode disaggregation isn't a silver bullet for LLM inference
kbesplv · reddit · 2026-10-29
A warmctl.io engineering deep-dive challenges the assumption that prefill/decode (P/D) disaggregation always wins, based on tests on a single 8×B300 node with vLLM nightly, DeepSeek-V4.1-Flash and expert parallelism (DEP8), scoring co-location efficiency as U+V.
Key findings:
- With expert parallelism, prefill on one rank stalls all eight GPUs (18.7 → 14.1 steps/s) — ranks synchronize at every MoE layer.
- Even 8 prompts/s of prefill traffic pushed U+V down to 0.84.
- Aligning prefill in bursts of 8 (one per rank) restored U+V to 1.00; agent-style turns (64K cached + 4K new tokens) hit 0.94–0.96 vs 0.77–0.81 unaligned.
- vLLM's --prefill-schedule-interval recovers 78% of the loss on a plain stream (+22% output tokens) for +370ms median TTFT.
- Co-location never beat parity on one GPU — phases took turns rather than overlapping.
Conclusion: splitting P/D on a single GPU type can't be cheaper; it still wins at rack scale, with different hardware per phase, or under hard inter-token-latency SLOs. Limits: one model, one engine, one node; real P/D with KV transfer and rack scale untested.
More from Infra
- Unsloth quants run 1T-param Mistral Large 4 on Mac Studio: 1-bit keeps 91% accuracy at 223GB — danielhanchen · 2026-10-29
- SMCI-backed neocloud Argentum AI seeks $5B credit facility for chip purchases — Beth_Kindig · 2026-10-29
- Microsoft returns to bond market seeking $25B+; AI debt wave lifts borrowing costs for all — rohanpaul_ai · 2026-10-29
- Self-hosted realtime avatar server: 30 FPS to 90 FPS on a free Colab T4 — SavingsWeather1659 · 2026-10-29
- WSJ: Apple refused flash deliveries when prices fell, leaving suppliers stuck — firstadopter · 2026-10-29
- Hyperscalers may cut out neoclouds as financing conditions worsen, analyst warns — RihardJarc · 2026-10-29