Cross-datacenter PDD for agentic LLM serving

机器之心 · wechat · 2026-07-20

Agentic infra hits the real bottleneck: inference latency

This long WeChat article explains why agent workloads change the economics of inference: users don’t wait for a single reply, they wait through many tool calls and rounds, so small per-step delays compound into minutes of degradation.

It introduces PDD (Prefill-RelayDecode-MainDecode), a cross-datacenter, heterogeneous PD-disaggregation design for LLM serving. The article argues that PD separation is necessary because prefill is compute-heavy while decode is memory/bandwidth-heavy, and that heterogeneous chips can each do what they’re best at.

Why cross-cluster matters

In real agentic traffic, prefix-cache hit rates can be very high, which makes cross-cluster transfer economically feasible.
The paper reports that in a 64K prompt scenario, same-datacenter PD separation has a P50 TTFT of about 4.2s, while moving decode to another city raises it to 6.7s.
At the tail, the uncapped cross-cluster setup reaches 18.3s P90 and 29.7s P99, which is close to the service ceiling for high-quality token serving.

How PDD works

P instance: runs prefill in cluster A.
RLD instance: stays in cluster A and starts decoding quickly over RDMA to hide network delay.
MD instance: sits in remote cluster B and takes over later.

KV cache is sent on two paths: fast local RDMA to RLD, and slower WAN/TCP to MD. RLD emits EagerOutputTokens while waiting for MD.

When MD finally receives the context, it does not need the full incremental KV cache. Instead, RLD sends only the token IDs, and MD recomputes the missing KV locally. The article says this keeps the handoff payload tiny and avoids unstable WAN transfer spikes.

Key measurements and claims

The handoff microbenchmark compares sending 4649KB of incremental KV cache over WAN versus sending 1.5KB of token IDs and recomputing locally.
WAN transfer is sometimes faster on average, but local recomputation is much more stable and avoids network jitter.
For long agent-style workloads, the authors say this can cut TTFT by 51.5% and single-token cost by 37.5% in their measured setup.

What the company is doing

The piece frames the company’s strategy as an “AgenticInfra” stack with three layers:

Compute aggregation: unify and schedule fragmented heterogeneous compute.
Token factory: optimize large-model serving efficiency.
AI productivity store: package infrastructure into industry solutions.

It also says the company is building infrastructure agents for ops, cluster maintenance, and operator generation, aiming for a closed loop where infrastructure becomes self-optimizing.

Original post →

Cross-datacenter PDD for agentic LLM serving

More from coding & agent