ThunderAgent merged into LLM-D: 2x agent serving throughput, TTFT from ~50s to 2-4s
simran_s_arora · x · 2026-11-04
ThunderAgent, a program-aware agentic inference system, has been merged into LLM-D. On real multi-turn agent traces (Claude Code / Weka), it delivers:
- 1.5x–2x output throughput boost
- Median TTFT dropped from 50s to 2–4s
- >2x reduction in agent serving cost
The design centers on session-aware routing, KV-cache affinity and per-Pod admission control, reusing KV state across turns via session management and endpoint affinity scoring.
Other milestones:
- Integrated into NVIDIA Dynamo 2.0, with the program abstraction as a first-class scheduling unit
- Integrated into SkyRL for agentic RL, accelerating SWE Agent rollout 3x on 40 H100s
- Accepted as ICML 2026 Spotlight (top 2.2%)
- MIT-licensed open source (471 stars)
More from coding & agent
- Concurrency limits are the hidden tax in voice AI — how Phonic solved it — graceisford · 2026-11-04
- Solo Dev Ships Claude Code-Built iOS Vocab App Where Claude Grades Answers at ~0.5¢ Each — haytchsquared · 2026-11-04
- Why AI agents optimize Go code better than Rust: domain expertise still steers the loop — unixterminal · 2026-11-04
- Alibaba's SPAR sales agent lifts conversion 15.8% in A/B test across 200k+ merchants — ycombinator · 2026-11-04
- 2637 prompts, 12.5B tokens, 383 PRs: designer opens up his entire AI-built portfolio process — tomjohndesign · 2026-11-04
- DeepSeek article outlines super-agent architecture: continuous, productive, proactive, persistent — LangChain · 2026-11-04