DeepMind-backed MKA benchmark crowns Claude Fable 5.2 at 57%, with 30x cost spread
ArtificialAnlys · x · 2026-11-11
Artificial Analysis partnered with Google DeepMind to launch the Multimodal Knowledge Agent (MKA) benchmark, testing agents on 200 private real-world visual tasks across 10 industries — schematic- and scan-heavy inputs producing slide decks, spreadsheets, PDFs, Word docs, and marked-up images.
- Scoring: 70 weighted rubric criteria per task, split into Regression (baseline) and Climb (hard) checks; headline metric is Climb WSR. Tasks run in the open-source Stirrup harness with code execution, web search/fetch, and image tools (up to 250 turns), graded by GPT-6 Luna (medium) as agentic judge
- Leaderboard: Claude Fable 5.2 (max) tops at 57% Climb WSR, Gemini 4 Carbon (high) at 54%, Claude Opus 5.5 (max) at 53%
- Cost spread: nearly 30x within 10 points of the top — Fable 5.2 averages $31.97/task vs $6.29 for Carbon and $1.11 for GPT-6.1 Sol; OpenAI models dominate the cost-performance Pareto frontier
- Verticals: no single winner — Anthropic leads 5 verticals, Google leads Science and Healthcare, Claude Sonnet 5.5 (max) narrowly leads Law & Government
Related event: DeepMind and Artificial Analysis Launch MKA Benchmark for Multimodal Agents(6 posts)→
More from Models
- Gemini 4 Carbon Arrives in Cursor, Tops CursorBench at 60.4% on High Effort — DynamicWebPaige · 2026-11-11
- Users criticize Anthropic's cyber safeguards for blocking vulnerability research — OwariDa · 2026-11-11
- Pelican test: Opus 5.5 uses 2x tokens of Fable 5.2 for only 4% more cost — and more bugs — PilgrimofHaqq2 · 2026-11-11
- Claude code-to-video threatens SeeDance and MiniMax pride: is video generation a detour? — teortaxesTex · 2026-11-11
- Pelican Test: Opus 5.5 uses 2x tokens of Fable 5.2 for only 4% more cost — and makes mistakes — PilgrimofHaqq2 · 2026-11-11
- Gemini 4 Carbon goes GA after two-week test, beats Fable 5.2 on some benchmarks at under half the cost — The Decoder · 2026-11-11