Infinigence's PDD Architecture Halves First-Token Latency, Cuts Costs by 37%
量子位 · wechat · 2026-07-30
At WAIC 2026, Infinigence unveiled its cross-cluster heterogeneous inference architecture, PDD. It connects local homogeneous data centers via low-cost WANs and evolves the traditional Prefill-Decode (PD) separation into a three-tier "P-RLD-MD" structure. This architecture overcomes KVCache transmission latency bottlenecks in Ethernet environments, reducing first-token latency by 51.5% and single-token costs by 37.5% in tests.
The team detailed three core insights driving the design: extreme hardware specialization, a 10x bandwidth reduction via prefix caching (DRC), and non-uniform latency caused by a tiny fraction of low-hit-rate requests in agent workloads. To solve this, PDD introduces a "relay" mechanism using local RLD instances to mask cross-cluster network delays. By transferring only Token IDs and recomputing locally, it converts unpredictable network transfers into deterministic computations, ultimately achieving a 37.5% improvement in cost-efficiency.
More from Infra
- China's AI boom drives up to 660% profit surges for domestic chip firms; GPU maker MetaX turns profitable — pstAsiatech · 2026-10-16
- Qdrant tests Perplexity's visual PDF embeddings: 21x smaller vectors, 99.9% after rescoring — jrxna · 2026-10-16
- When Does an Agent Actually Need an API Gateway? Reddit Weighs Direct Calls vs. Unified Endpoints — TurbulentStiffness · 2026-10-16
- Taiwan to build $4B AI manufacturing complex in Texas to back Nvidia and TSMC — Polymarket · 2026-10-16
- When the KV Cache Is Slower Than the GPU: Remote KV Tiers Backfire in Production — MultiverseComputingCAI · 2026-10-16
- GitHub rewrote the 800k-line Copilot runtime in Rust using AI agents, cutting CPU and memory by over 90% — bibryam · 2026-10-16