Together Compiler: agents re-tune inference for up to 2.6x TPS gains
simran_s_arora · x · 2026-11-05
Together AI launched Together Compiler (TKCC), an agent-driven system that continuously re-optimizes inference as production traffic changes.
How it works
- Persistent serving engine: starts once, agent-written forward passes hot-reload without restarts — 252x more experiments per run
- Two-tier evaluation: a 12-layer fast tier screens candidates, a 60-layer slow tier confirms gains on real traffic
- Raw engine source let agents speed up Kimi K3's MLA kernel 1.7–2.5x
Results (self-reported)
- Up to 2.6x TPS over best published InferenceX configs, under 24 hours per model
- MiniMax M3.1 Flash on B200 hit 1,111 P90 TPS/user (1.8x vs 633 baseline)
- Counterexample: GLM 5.3 on MI355X only improved 463 → 480 TPS (4%)
Live on Together serverless endpoints, limited beta for dedicated endpoints, not open source. Tested on B200/B300/GB300/H100/MI355X/Trainium2.
More from coding & agent
- Dev builds terminal harness SUTRA for models, DeepSeek session costs just $0.0001 — cneuralnetwork · 2026-11-05
- Running Claude Code from a phone: 60% of a day's work done in 30 minutes — PrajwalTomar_ · 2026-11-05
- OpenClaw robotic arm works as release manager, cherry-picking thousands of PRs — vincent_koc · 2026-11-05
- One prompt, one playable 3D cyberpunk racing game: hands-on with Tencent's Hy4 — HeyAmit_ · 2026-11-05
- Tim O'Reilly at AgentCon: open architecture beats open-source licenses for the agentic web — bibryam · 2026-11-05
- Dev tip: install GitHub CLI for agent dev so agents can see issues, PRs and CI — DanielLockyer · 2026-11-05