Together AI's TKCC lets agents hot-patch inference engines: 252x more experiments, 2.6x throughput
simran_s_arora · x · 2026-11-07
Together AI published "Together Compiler (TKCC): Continual Compilation for AI Inference," letting coding agents continuously modify and test inference engine code without restarts, measured on real workloads.
The core challenge: agent-driven inference optimization faces a throughput-vs-fidelity tradeoff — offline kernel benchmarks are unreliable due to data-dependent behavior (e.g., MoE), while end-to-end engine optimization suffers from slow launch times.
Approach and results:
- Tiered Persistent Engine decouples engine launch overhead from forward-pass computation
- A cost model picks parallelism configurations to seed optimization
- Extensive ablations on agent context management, plus production deployment insights
- 252x more agent experiments, up to 2.6x higher throughput than the best open-source engines on frontier models, and 1.7x over hand-written Megakernels on Llama 1B
More from coding & agent
- Amazon now blocking personal AI agent shopping traffic from home IPs — unixterminal · 2026-11-07
- Claude Duet adds Claude Code sessions: approve changes and send updates from your phone — lydiahallie · 2026-11-07
- A 4B model trained on one GPU in 2 hours beats Jev 1.13 at agent decisions, 86.5% vs 80.6% — ivan_bezdomny · 2026-11-07
- Three ways to build an AI app: single agent, agent team, or graph workflow — rseroter · 2026-11-07
- 10,000 agents playing Minecraft spin up in 60s on 200×48-core machines — KlausCodes · 2026-11-07
- Curated Bend 2 package ecosystem tracker snapshots 1,036 packages on BendHub — SubtleGradient · 2026-11-07