Together Compiler: Agents That Continuously Optimize Kernels, Cutting Latency Up to ~Half
togethercompute · x · 2026-11-07
Together launched Together Compiler (TKCC), the first product of its "Polymorphic Inference" push: agents that continually optimize kernels for your specific model, hardware, and traffic shape.
Why it exists
- Inference workloads are highly dynamic: none of OpenRouter's top-10 open-source LLMs share an architecture; Together serves trillions of tokens daily with wildly varying batch sizes, sequence lengths, MoE expert imbalance, and attention sparsity across H100/B200/B300/NVL72/Groq hardware — manual optimization no longer scales.
- Agents fit the search problem in theory but are bottlenecked by experiment speed.
Results so far
- +12% to +20% median TPS on serverless endpoints researchers had already heavily hand-optimized.
- Production customers cut response and generation times by roughly a third to nearly half.
Built to scale frontier kernel research (FlashAttention, ThunderKittens, ThunderAgent teams) to every customer deployment; 40+ models already in production.
Related event: Together AI launches agentic continual compiler TKCC for faster inference(4 posts)→
More from coding & agent
- OpenAI ships API starter projects for agents, voice, and more, remixable in Codex — OpenAIDevs · 2026-11-07
- OpenAI Developers plugin adds Showcase tab; Pro/Business subscribers can claim API credits — OpenAIDevs · 2026-11-07
- Google resets everyone's Gemini and Argon quota on Antigravity ahead of the weekend — rseroter · 2026-11-07
- GTEx MCP Server gives AI assistants 25 tools to query genomics across 54 tissue types — modelcontextprotocol · 2026-11-07
- Google launches MCP connector for Cloud Composer, enabling natural-language workflow control — modelcontextprotocol · 2026-11-07
- AllMCPs launches an agent-first MCP directory with automated uptime and security checks — Aliceable · 2026-11-07