HYPIC: Cache Acceleration for Hybrid Attention Models
小红书技术REDtech · wechat · 2026-07-16
The LLM inference teams from Xiaohongshu, Peking University, and Shanghai Jiao Tong University proposed HYPIC to solve the incompatibility between hybrid attention LLMs and Position-Independent Caching (PIC) in RAG/Agent long-context scenarios.
Core Approach
- Linear Layers: Instead of only caching the "zero-initial-value terminal state", it additionally caches the segment cumulative transition operator TC. During reuse, states can be combined in constant time to almost perfectly reconstruct results for any prefix.
- Full Attention Layers: Introduces a tiny seam window (default 8 tokens) for local recomputation to fix cross-segment attention biases.
- Segment Parallelism: Splits cold requests into multiple fragments for parallel prefill, then aggregates results, reducing TTFT for long cold requests.
Experimental Results
- Across 4 production-grade hybrid attention models and 5 workloads, HYPIC reduced TTFT by 3.25x on average compared to PrefixCache.
- Under the same SLO, sustained QPS increased by 1.66x.
- Task quality differs by an average of only 1.71 points compared to full recomputation; a naive "direct addition" approach loses significant accuracy.
- On a pure cold request path, TTFT for a 32k token request across 8 workers dropped from 2.83 seconds to 0.49 seconds.
Engineering Implementation
- Implemented based on SGLang, comprising about 14,000 lines of Python/Triton.
- Tested on Ring-mini/flash-linear-2.0, Qwen3.5-35B-A3B/122B-A10B, HotpotQA/TriviaQA/MultiNews/GovReport, and a production RAG trace.
More from Infra
- Why a 1GW Chinese AI data center may be plausible after all — teortaxesTex · 2026-07-22
- LFM2.5-8B-A1B doubles its tokenizer vocab and cuts on-device decoding time up to 3.7x — maximelabonne · 2026-07-22
- China’s AI arms race is increasingly defined by chips, data centers, and open models — BenBajarin · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- Gavin Baker argues Nvidia may be one of open source AI’s biggest supporters — GavinSBaker · 2026-07-22
- AI Power Demand Exposes US Energy Gap, Urging Shift from Scarcity to Abundance — bradneuberg · 2026-07-22