Artificial Analysis details benchmark methodology: every 6 hours, sustained load, agentic sessions
ArtificialAnlys · x · 2026-11-05
Artificial Analysis published the methodology behind its API performance benchmarking:
- Workloads: single requests at several prompt lengths (short to long context), plus multi-turn agentic sessions from coding and knowledge-work evals with tool calls
- Production-like load: each scheduled run serves fixed single-request workloads alongside one rotating workload, which starts last and runs under full load
- Frequency: every 6 hours; rotating long-context and multi-turn workloads measured once daily; deprecated-model endpoints every 18 hours and their rotating workloads every 3 days
- Caching: cache hit rates measured in multi-turn sessions, updated monthly from the trailing four-week median
- Longer prompts raise time to first token and lower output tokens per second
More from Models
- Ripple claims $32M compute for open-weight coding SOTA; skeptics question the math — teortaxesTex · 2026-11-05
- Gemini 4 Flash (codename Hydrogen) spotted A/B testing in AI Studio and LMArena — lyraxana · 2026-11-05
- Perceptron's new 3D hand model beats ACE-Ego-Hand by 12-25% on HOT3D and ARCTIC — gui_penedo · 2026-11-05
- 0.8B open-source HaloGuard 2.0 claims streaming-guard SOTA, beating models 5-10x its size — const_reborn · 2026-11-05
- MolmoAct 2 Takes Second to GPT-6 Astra in RoboColosseum's Real-World Robot Tests — DJiafei · 2026-11-05
- Claude Max subscriber at $100/month says Opus 5.5 'almost feels unlimited' — prasenx · 2026-11-05