FULL STORY

Together AI's Compiler: From Launch to Validated Speedups

Together AI launched Compiler, claiming up to 252x throughput gains for agentic inference. SemiAnalysis then validated the results and endorsed it as production-ready.

2026-11-05 ~ 2026-11-06 · 2 episodes · 27 posts

Episode 1 · Together AI Unveils TKCC, an Agentic Continuous Compiler for Inference (2026-11-05, 25 posts)

On November 5, Together AI's research team released the Together Compiler (TKCC). According to research lead simran arora, this is the first detailed study of production-scale "agentic inference optimization," accompanied by a paper and blog post. The system targets the "polymorphism" problem of inference workloads: on OpenRouter, 80% of traffic goes to models released in the past three months, the top ten popular open-source models each use different architectures, and traffic patterns shift every second (users gradually migrating to new models, sudden fluctuations in request volume and types). Traditional static optimization is obsolete, hence agent-driven continuous compilation.

Confirmed

  • Final results: building on the best current InferenceX engine configuration, TKCC set a new record for concurrent 1 TPS (per-user P90 throughput) across all tested scenarios. The pipeline first uses parallel cost models to filter valid parallel configurations, then up to 2 nodes explore and deliver peak performance within 24 hours.
  • Experimental efficiency improved 252x over previous approaches; MiniMax M3.1 reached 1111 TPS.
  • Three key designs: hierarchical persistent engines (the engine starts once and can hot-reload any candidate forward pass without restarts), parallel cost models, and supporting mechanisms for the throughput-fidelity trade-off.
  • Technical lineage: the team's earlier work on AI acceleration frameworks like ThunderKittens and Megakernels aimed to scale per-customer manual optimization into an automated pipeline delivering peak performance within hours.

Why it matters

  • As new models ship faster, architectures diverge, and traffic patterns shift by the second, inference providers can no longer rely on manual or static tuning. TKCC demonstrates that agent-driven compilation is viable at production scale and sets throughput records under tight compute constraints (up to 2 nodes, 24 hours), offering direct reference value for inference cost and performance optimization.

5 more related posts →

Episode 2 · SemiAnalysis Validates Together Compiler TKCC as Production-Ready (2026-11-05, 2 posts)

SemiAnalysis validated Together Compiler (TKCC) from Simran Arora's team, praising it as one of the coolest things, with continuous automatic optimization that is production-ready.