Together AI Deep Dive: Autoscaling Endpoints for LLM Inference
togethercompute · x · 2026-08-01
Together AI published a deep technical article on autoscaling endpoints for LLM inference. It points out that traditional CPU-style metrics fail to capture the full picture of GPU pressure; a GPU might show 60% busy while the engine's queue is already backing up.
The article explores core engineering practices:
- Scaling Metrics: Out of eight available metrics, why "in-flight requests" is recommended as the default.
- Window Tuning: How to configure up and down windows, explaining the sawtooth replica count trap that almost every engineer hits.
- Cold Start Costs: Phase-by-phase measured numbers of what a cold start actually costs on a 1xH100.
It also includes an experiment replaying one load profile under three different policies, two of which failed to scale entirely.
More from Infra
- a16z: AI Infrastructure Demand Shows No Signs of Slowing Amid Supply Chain Snags — a16z · 2026-08-01
- Vercel AI Gateway Adds Team and Project Spend Budgets — cramforce · 2026-08-01
- Tesla Signs 469MW Solar Deals to Lock in AI Compute Power Years Ahead — XFreeze · 2026-08-01
- Local Deployment on DGX Spark: Exploring Upgrades Beyond Qwen 3.5 122B — Voxandr · 2026-08-01
- Analyst Spots Equinix Expanding San Jose Campus by ~200MW — BenBajarin · 2026-08-01
- Micro Center Reports Major Price Hikes for Nvidia RTX 5090 — soumitrashukla9 · 2026-08-01