SJTU's TAO Cuts LLM Agent Serving Latency Up to 26.8% With Tool-Aware Joint Scheduling
AIOR-Research · hf · 2026-10-29
Shanghai Jiao Tong University and Stanford propose TAO, a tool-aware joint scheduling framework for LLM-based agentic serving. Its value interface combines near-term tool readiness, time-decayed node-specific cache credit, and token footprint, with a dynamic program placing sessions under capacity constraints. Across 12 model-workload-concurrency settings, TAO cuts latency by up to 26.8%, TTFT by up to 36.3%, and boosts decode throughput by up to 29.5% over the best baseline per metric.
More from coding & agent
- TTME beats test-time training on math discovery with 84% fewer FLOPs — rohanpaul_ai · 2026-10-29
- 10 open-source vector DBs for AI agents, from Milvus to sqlite-vec — victor_explore · 2026-10-29
- Claude agent builds its own picture tool and sets its own avatar — nathanborror · 2026-10-29
- Forbidding AI comments: one dev's rules for slop-free Claude Code — seatedro · 2026-10-29
- Naver MCP Server brings Naver OpenAPI search for blogs, news and images to MCP clients — modelcontextprotocol · 2026-10-29
- Clinicaltrials MCP connector lets AI agents search ClinicalTrials.gov by drug, phase and status — modelcontextprotocol · 2026-10-29