Artificial Analysis Overhauls API Benchmarks: Time per Task Replaces Streaming Speed
ArtificialAnlys · x · 2026-11-05
Artificial Analysis revamped its API performance benchmarking across 1,000 serverless endpoints. Key changes: Time per Task is now the headline metric (a fast-but-verbose model can lose to a slower concise one); speed is measured end-to-end including first-token wait, with token counts from provider-reported APIs; benchmarks now include agentic multi-turn workloads; and API-default effort is tracked alongside max effort.
Takeaways:
- OpenAI models dominate the Intelligence-vs-Time Pareto frontier: GPT-6 Astra finishes tasks in 17 min using 50k output tokens, vs 26 min for Claude Opus 5.5 and 29 min for Gemini 4 Carbon (which burn 120k–170k tokens).
- Default (medium) effort cuts Time per Task 3–5x and cost 2–4x: Opus 5.5 drops from 26 min/$10.04 to 6.8 min/$2.45 for only 5–11 fewer Intelligence Index points.
- Decode-speed-optimized endpoints fall hardest under the new metric, since queueing and burst delivery inflate first-token latency.
Related event: Artificial Analysis Revamps API Benchmarks Around Time per Task(2 posts)→
More from Infra
- LFM3-Omni-2B ported to run fully offline on iPhone at ~80 tok/s with custom WebGPU kernels — JosephJacks_ · 2026-11-05
- Orbital Data Centers Launch to Sun-Synchronous Orbit, Memed as Elves Sailing West — rand_longevity · 2026-11-05
- Vietnam's Lâm Đồng to bring Starlink to mountain areas where poor signal hampers rescue — elonmusk · 2026-11-05
- SpaceX partners with Qualcomm for nationwide Starlink Mobile 5G network from 2027 — Polymarket · 2026-11-05
- Stopping runaway agent spend before the bill: pre-call guardrails and three gaps found the hard way — MaverikSh · 2026-11-05
- US AI scale faces two physical gates: delivered power and manufacturing — ingliguori · 2026-11-05