Baseten's WavePP: up to 2.9x prefill throughput on same GPUs, same kernels

baseten · x · 2026-11-20

Baseten introduced WavePP, a pipeline-parallelism (PP) prefill runtime built on TensorRT-LLM. Each PP stage runs as an independent lightweight model worker with its own KV cache and request prep; WavePP overlaps request admission with pipeline execution — asynchronously finding a prefix reusable across all stages, protecting cached state, reserving space for remaining input, and dynamically planning chunk sizes to maximize pipeline fill.

Results:

Original post →

More from Infra

Infra channel →