Together AI: leaderboards are for shortlisting, your production traffic is the real benchmark
togethercompute · x · 2026-11-09
Together AI argues that public benchmarks are only useful for building a model shortlist — the real test is replaying your actual production workloads (long-in/short-out, cached requests, tool calls, repeated agent turns) against candidates and seeing which models pass your own evals. Their example diagram shows the leaderboard's top model failing internal evals while the #3 model ships. "Leaderboards are for discovery; your traffic is for decision-making."
More from coding & agent
- Simon Willison hides AI PR descriptions in details/summary tags — Simon Willison · 2026-11-09
- Team's AI agent fixes 40 of 41 user-reported issues in a single weekend — julianweisser · 2026-11-09
- HEELO Harness Framework Lifts Terminal-Bench Resolution from 37.3% to 55.5% — Haibo Jin · 2026-11-09
- NVIDIA's AgentBaton: Well-Timed Strong-to-Weak Model Handoffs Slash Agent Costs — nvidia · 2026-11-09
- FrontierCS 2 update: Claude Sonnet 5.5 beats reference on 50% of tasks, reward hacks published — ajratner · 2026-11-09
- "Being technical" now means having the patience to ask Claude until you get it — brandon_galang · 2026-11-09