vLLM Splits Prefill and Decode, TileRT Takes Over Decode
vllm_project · x · 2026-07-15
The vLLM team shared a specific use case: after decoupling vLLM's prefill and TileRT's decode, the decode side can be hot-swapped based on the workload.
Key Points
- The TileRT team integrated this via the vLLM V1 connector interface, requiring no changes to vLLM code.
- It defaults to vLLM's decode to ensure throughput; for scenarios prioritizing low latency, like agents or real-time assistants, it can switch to TileRT decode.
- Both sides maintain the same OpenAI-compatible interface and prefix caching; switching is essentially just a routing change.
Performance Data
- TileRT reported that on GLM-5.1-FP8, using 8× B200 + MTP, single-user decode reached around 618 tok/s.
- This is roughly 2 times faster than its baseline without MTP.
- At peak acceptance rates, speed approaches 800 tok/s.
The authors thanked TileRT and inferact for their collaboration.
Related event: vLLM and TileRT Introduce Decoupled Inference Stack(3 posts)→
More from Infra
- Nvidia Rubin is coming, pointing to the next AI compute platform — ezyang · 2026-07-21
- Tesla’s FSD v14 Lite is reportedly headed to 4 million older HW3 cars — MatthewBerman · 2026-07-21
- TSMC’s 3nm utilization reportedly tops 120% as AI demand drives a $190B capex cycle — tengyanAI · 2026-07-21
- Nativ brings local AI model running to Mac with a desktop app and localhost API — Simon Willison · 2026-07-21
- Octen says agent search now runs at 62ms P50 with only a 6ms P90 gap — aakashgupta · 2026-07-21
- Zhipu acquires a compiler-team spinout to optimize AI inference on domestic chips — zephyr_z9 · 2026-07-21