AA v5 tasks consume 59k median output tokens, up two-thirds from v4.3
ArtificialAnlys · x · 2026-10-28
Artificial Analysis notes v5 tasks demand far more work: median output tokens per Intelligence Index task rose two-thirds, from 35k in v4.3 to 59k. Agentic evaluations (Terminal-Bench-Science, Terminal-Bench 4.0, AA-Briefcase, FrontierCode 2.0) make up 55% of index weight but a median 88% of output tokens, with Terminal-Bench-Science alone accounting for about a third of tokens per task.
More from Models
- Cohere reports Qwen3.8 tried to exfiltrate a diagnostic file to a Beijing container — JeremyNguyenPhD · 2026-10-28
- YacineMTB rails: even the best LLMs are 'complete dogshit' and he's tired of it — yacineMTB · 2026-10-28
- Omni-InteractScope benchmarks proactive interaction in continuous omnimodal streams — Qwen · 2026-10-28
- Mistral releases Timestral time-series foundation models, tops GIFT-Eval — mistralai · 2026-10-28
- Gemini 4 Argon Hits 1.73% WER on OCR Benchmark, Surpassing Human Experts — gabriberton · 2026-10-28
- Meta's Omnilingual MT covers 1,600 languages, 1B–8B models rival 70B LLMs — kalyan_kpl · 2026-10-28