Together Compiler: agent-driven continual compilation, 252x more experiments
simran_s_arora · x · 2026-11-05
Together AI's research team introduces the Together Compiler (TKCC), a production service already deployed across parts of their inference fleet that uses agents to continuously discover the most efficient inference runtimes for any combination of workload, model, and silicon.
Key points:
- Motivation: 80% of OpenRouter traffic goes to models released in the last three months, and each of the top ten open models uses a different architecture—static inference engines with hand-tuned configs can't keep up.
- Concept: a shift from Static Inference to "Polymorphic Inference"—systems that continuously and autonomously adapt to workloads, hardware, and models for better utilization and economics at scale.
- Results: agents can run 252x more optimization experiments per hour than before, with TKCC continuously optimizing the production inference stack.
More from Infra
- J.B. Huang's new video explains how LLM serving actually works, from fast to cheap — jbhuang0604 · 2026-11-06
- US BIS extends suspension of affiliates end-user rule through January 10, 2027 — pstAsiatech · 2026-11-06
- Meta's Axia Learns Scaling Laws for Ranking Models from a Few Trial Runs — _reachsumit · 2026-11-06
- US trade deficit ex-data centers is just 0.7% of GDP, lowest since 2000 — soumitrashukla9 · 2026-11-06
- Modal's GPU Glossary praised as an excellent free reference — eigenron · 2026-11-06
- Underdog Woof: 31.7MB open-weight TTS model claims 1.42% WER, runs fully in-browser — _sathvikr · 2026-11-06