AWS launches SageMaker JumpStart Model Leaderboard comparing 70+ models on quality, speed, cost
AWS ML Blog · rss · 2026-11-19
AWS launched the SageMaker JumpStart Model Leaderboard in SageMaker Studio, comparing 70+ open-weight models (Meta, Google, Qwen, IBM, Arcee and more) side by side. AWS runs BIG-Bench Hard, GPQA Diamond, and MMLU-Pro itself with one harness for comparable quality scores, and measures TTFT, throughput, and hourly instance cost by sending standardized workloads to real endpoints. You can switch use-case profiles and optimization goals — Balanced, Cost, Latency, or Throughput optimized (speculative decoding available) — then go from filters to a deployed model via a step-by-step walkthrough. Cost estimates cover SageMaker instance rates only.
More from Infra
- OpenAI: median researcher spends $2,400/day on agents, token costs now 2x labor costs — daveholtz · 2026-11-19
- Fchollet: latent token demand is near-infinite, cost is the only brake as RoI keeps improving — fchollet · 2026-11-19
- Running Hy4-Flash locally on 2x RTX 3090: Q3 quant hits PP 63 / TG 22 tokens per second — EmilPi · 2026-11-19
- Meta exec: without owning compute on US soil, economic and national security foundations crumble — BenBajarin · 2026-11-19
- OpenAI adds discounted background mode: save 30-40% per token on GPT-6.2 with 2h/8h windows — OpenAIDevs · 2026-11-19
- EchoMoE Sends Only Deltas Between Nodes, Cutting MoE Inference Traffic 58% — sivareddyg · 2026-11-19