Huawei's openJiuwen Optimizes Agent KVCache, Cutting First Token Latency by 57%
机器之心 · wechat · 2026-08-11
Huawei's openJiuwen open-source platform introduced a "compute-affinity" coordination mechanism to solve KVCache memory waste and scheduling latency during multi-Agent concurrent long tasks.
- The Disconnect: Traditional inference engines rely on generic cache policies (like LRU) and are blind to Agent task states (e.g., waiting for tools, sub-task completion), causing memory hoarding or redundant recomputations.
- AgentHint Contract: The framework actively sends semantic hints to the inference engine during state changes, upgrading cache management into proactive eviction, offloading, and prefetching.
- Three-Layer Synergy: JiuwenSwarm tracks task states; SAM (Session-Aware Manager) fine-tunes local NPU HBM allocation; SPM (Pooling Manager) leverages the Ascend Lingqu bus for high-speed cross-device cache migration (HBM/DDR/SSD).
- Benchmarks: On concurrent SWE-bench tasks, enabling this mechanism reduced Time-To-First-Token (TTFT) by 57.46%, end-to-end latency by 27.61%, and peak pooled cache usage by 25.24%.
More from coding & agent
- Building a Ground-Up AI E-commerce Creative System: Mockups, Evals, and Agents — aishashok14 · 2026-08-11
- Hands-on with Claude Code: Auto-generating Follow-up Task Cards — dotey · 2026-08-11
- Reconstructing 3D Scenes as Code: Agent Workflow for Real-time Editing — joemeno · 2026-08-11
- BlackHat Warning: 171 Ways for AI Agents to Escape Sandbox via Communications — Ghost_Pilot_MD · 2026-08-11
- Stateful Agent Swarms on Low Compute: A 20x Productivity Boost — wonderwomancode · 2026-08-11
- I Talked a Client Out of an AI Agent for a No-Code Website Builder — amiitk · 2026-08-11