Artificial Analysis launches Endpoint Accuracy Index v1.1: nearly 1 in 4 serverless endpoints score below reference
ArtificialAnlys · x · 2026-11-06
Artificial Analysis released Endpoint Accuracy Index v1.1, adding coverage for GLM-5.3 Flash and Muse Spark Open. Across 100+ benchmarked serverless endpoints, nearly 1 in 4 scores below its self-hosted reference deployment.
- Method: scores each endpoint as a percentage of a reference deployment, equally weighting tool calling (BFCL-500), scientific reasoning (HLE-250), and long-context recall (AA-LCR-25). Providers quantize weights, custom kernels, cap outputs, or ship bugs to optimize speed/cost
- v1.1 changes: AA-LCR upgraded to v1.1 (fixed answer-key errors, graded by GPT-5.6 Luna), repeats raised from 10 to 25, tightening the 95% CI from ±4.7 to ±2.6; six open-weight models now covered
- GLM-5.3 Flash: 16 of 22 endpoints statistically in line, one above reference (104.2); two endpoints with 32K output caps truncate half of HLE-250 responses, scoring 22%/29% vs 55%; two under-reason 30% fewer tokens and score 48%
- Muse Spark Open: 3 of 5 endpoints match (99.5–99.9, one serving FP4); the other two lose accuracy only on tool calling, returning text-defined tool arguments as numbers (50%/51% on BFCL-500 vs 59%)
More from Infra
- Is AMD 7900 XT/XTX viable for video generation in late 2026? — Beneficial_Fish_7509 · 2026-11-06
- Blue Origin unveils Northstar engine: 715,000 lbf thrust, 30% lighter than BE-4 — beffjezos · 2026-11-06
- DFlash drafts whole token blocks in parallel to fix speculative decoding bottleneck — Lianhuiq · 2026-11-06
- A single desktop B70 runs an entire AI livestream radio, generating music in real time — prince_pringle · 2026-11-06
- Inferact, founded by vLLM creators, now serves 7 trillion tokens a day — Xianbao_QIAN · 2026-11-06
- Q-CTRL launches Dynamics Simulator Qiskit Function on IBM Quantum — MJBiercuk · 2026-11-06