Artificial Analysis details Search Index methodology: open-source Stirrup harness
ArtificialAnlys · x · 2026-10-30
Artificial Analysis shared methodology details for its Search Index leaderboard: all Search API rows run on its open-source agent harness "Stirrup," with DeepSeek Search settings documented and data current as of Oct 19, 2026. The score is an equal-weighted mean of DeepSearchQA F1, BrowseComp exact-answer accuracy, and AA-Omniscience accuracy, alongside cost-per-task and time-per-task metrics. Providers can contact them for inclusion.
More from Models
- Claude Opus 5.5 parody music video 'LGTM' hits 1.7M views in three weeks — cgarciae88 · 2026-10-30
- HF's Open TTS Leaderboard adds streaming tab: pocket-tts hits 12ms TTFA — Acceptable-Cycle4645 · 2026-10-30
- VLMs hide spatial structure in their representations, and training can preserve it — jiqizhixin · 2026-10-30
- Cognition benchmark: Gemini 4 Carbon scores 40% near-SOTA at $4.04 per task — scaling01 · 2026-10-30
- Users beg OpenAI to fix AI writing's predictable patterns: 'give it a soul' — createthiscom · 2026-10-30
- OpenAI's Astra Becomes Top Distillation Target for China's AI Labs — pstAsiatech · 2026-10-30