Meituan's custom SLM inference engine hits 10x SGLang throughput at sub-20ms latency with 67% MFU

美团技术团队 · wechat · 2026-10-15

Meituan's tech team redesigned online inference for bounded-generation SLM workloads (0.5–1B models) in search and recommendation.

Problem: Search/rec SLM requests are Prefill-Only or fixed-step BeamSearch with bounded iteration counts, but SGLang/vLLM are built for open-ended generation—their scheduling is a negative optimization here, and request arrivals deviate sharply from Poisson, causing low GPU utilization with high latency.
Architecture: A static DAG organizes the complete business request as the unit of scheduling, measurement and capacity planning; models compile offline via AOTInductor into AOTI artifacts, while a C++ runtime handles scheduling and concurrency, eliminating Python dispatch overhead.
Optimizations: PackedPrefill over valid tokens only, intra-request shared-prefix K/V, fixed BeamSearch compilation, selective FP8 matmuls; cross-request prefix cache is dropped due to near-zero hit rates.
Results: At <20ms average end-to-end latency (P99 70ms), Prefill-Only hits 67% effective MFU and 10x SGLang throughput (1,500 sample/s for a 0.6B model); short-prompt BeamSearch reaches 41% MFU, 6x SGLang (300 request QPS).
Deployment: Already validated in Meituan's search intent routing, relevance judging and generative recall, with full rollout at controlled inference cost.

More from Infra