Together Compiler uses agents to continually optimize AI inference, hitting 252x throughput gains
simran_s_arora · x · 2026-11-05
Together AI's simransarora unveiled the Together Compiler, a 'continual compilation' system for AI inference. Key insight: inference workloads are polymorphic — 80% of OpenRouter traffic goes to models released in the last three months, and the top ten open models each use different architectures. The system uses agents to re-optimize deployments as usage shifts. Highlights: TKCC lets agents run 252x more optimization experiments per hour than prior approaches, and the team found surprising cases where human inductive biases limited what agents could achieve. SemiAnalysis's Dylan Patel praised it as advancing agent-expert collaboration for system optimization.
More from Infra
- Cloudflare now auto-retries Durable Object calls up to 4 times, no retry loops needed — dinasaur_404 · 2026-11-06
- Nasdaq sheds $3.2tn in worst day since April 2025 as Nvidia sinks 9.3% on data center fears — rohanpaul_ai · 2026-11-06
- J.B. Huang's new video explains how LLM serving actually works, from fast to cheap — jbhuang0604 · 2026-11-06
- US BIS extends suspension of affiliates end-user rule through January 10, 2027 — pstAsiatech · 2026-11-06
- Meta's Axia Learns Scaling Laws for Ranking Models from a Few Trial Runs — _reachsumit · 2026-11-06
- US trade deficit ex-data centers is just 0.7% of GDP, lowest since 2000 — soumitrashukla9 · 2026-11-06