Together AI Launches Together Compiler, an Agentic System That Optimizes Inference in Hours and Sets New Throughput Records
On November 5, Together AI's research team released the Together Compiler (TKCC). According to research lead simran arora, this is the first detailed study of production-scale "agentic inference optimization," accompanied by a paper and blog post. The system targets the "polymorphism" problem of inference workloads: on OpenRouter, 80% of traffic goes to models released in the past three months, the top ten popular open-source models each use different architectures, and traffic patterns shift every second (users gradually migrating to new models, sudden fluctuations in request volume and types). Traditional static optimization is obsolete, hence agent-driven continuous compilation.
Confirmed
- Final results: building on the best current InferenceX engine configuration, TKCC set a new record for concurrent 1 TPS (per-user P90 throughput) across all tested scenarios. The pipeline first uses parallel cost models to filter valid parallel configurations, then up to 2 nodes explore and deliver peak performance within 24 hours.
- Experimental efficiency improved 252x over previous approaches; MiniMax M3.1 reached 1111 TPS.
- Three key designs: hierarchical persistent engines (the engine starts once and can hot-reload any candidate forward pass without restarts), parallel cost models, and supporting mechanisms for the throughput-fidelity trade-off.
- Technical lineage: the team's earlier work on AI acceleration frameworks like ThunderKittens and Megakernels aimed to scale per-customer manual optimization into an automated pipeline delivering peak performance within hours.
Why it matters
- As new models ship faster, architectures diverge, and traffic patterns shift by the second, inference providers can no longer rely on manual or static tuning. TKCC demonstrates that agent-driven compilation is viable at production scale and sets throughput records under tight compute constraints (up to 2 nodes, 24 hours), offering direct reference value for inference cost and performance optimization.
2026-11-05 ~ 2026-11-05 · 12 related posts
Primary sources
- Together Compiler optimizes new open models for production in hours, not weeks — togethercompute ·
- Together Compiler: agentic compilation hits 252x more experiments, 1111 TPS on MiniMax M3.1 — simran_s_arora ·
- Together Compiler sets new concurrency TPS records across models with just 2 nodes and 24 hours — simran_s_arora ·
- [source] Together Compiler: agentic compilation hits 252x more experiments, 1111 TPS on MiniMax M3.1 — simran_s_arora · 2026-11-05
- From static to polymorphic inference: Together's case for continual compilation — simran_s_arora · 2026-11-05
- Agent-based kernel optimization faces a fidelity-throughput tradeoff: random data misleads, cold starts cost 12.7 minutes — simran_s_arora · 2026-11-05
- How the Together Compiler works: persistent engines, agent message boards, and a parallelism cost model — simran_s_arora · 2026-11-05
- [source] Together Compiler sets new concurrency TPS records across models with just 2 nodes and 24 hours — simran_s_arora · 2026-11-05
- Inside the Together Compiler: from Megakernels and PTX hacking to agent-driven optimization — simran_s_arora · 2026-11-05
- Together ships first detailed research on agentic inference optimization at production scale — simran_s_arora · 2026-11-05
- Together Compiler debuts: continuous kernel optimization for the polymorphic inference era — simran_s_arora · 2026-11-05
- Together AI ships TKCC: 1200+ TPS on DeepSeek v4.1 Flash with 2 nodes in 24 hours — simran_s_arora · 2026-11-05
- [source] Together Compiler optimizes new open models for production in hours, not weeks — togethercompute · 2026-11-05
2 near-duplicate retellings: togethercompute · simonguozirui