Google DeepMind and Artificial Analysis launch MKA benchmark; Claude Fable 5.2 tops at 57% Climb WSR
ArtificialAnlys · x · 2026-11-11
Artificial Analysis partnered with Google DeepMind to launch the Multimodal Knowledge Agent (MKA) benchmark, testing agents on 200 private real-world visual tasks across 10 industries — reviewing schematics, architectural drawings and clinical scans to produce slide decks, spreadsheets, PDFs, Word docs and marked-up images.
Design: Each task is graded against 70 weighted expert rubric criteria split into Regression (baseline) and Climb (hard, separating) checks, with headline score Climb WSR. Tasks run in the open-source Stirrup harness with code execution, web tools and image viewing (up to 250 turns), judged by GPT-6 Luna (medium).
Results
- Claude Fable 5.2 (max) leads at 57% Climb WSR, ahead of Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%; scores span 28–57%
- Cost per task varies 30× within 10 points of the top: Fable 5.2 averages $31.97 vs $6.29 for Gemini 4 Carbon and $1.11 for GPT-6.1 Sol; four OpenAI models sit on the cost-performance Pareto frontier
- No single model dominates all verticals: Fable 5.2 leads five, Opus 5.5 leads Education and Construction, Gemini 4 Argon leads Science, Gemini 4 Carbon leads Healthcare
Related event: DeepMind and Artificial Analysis Launch MKA Benchmark for Multimodal Agents(6 posts)→
More from Models
- Google Antigravity ships 81 weekly updates: Gemini 4 Carbon and /swarm multi-agent land — rseroter · 2026-11-11
- Rumor: OpenAI to release "the big one" under competitive pressure — iruletheworldmo · 2026-11-11
- Study: Half of ChatGPT and Gemini stock picks concentrate in just 10 stocks — alejandroll10 · 2026-11-11
- Ethan Mollick: Gemini 4 Carbon shows Google is back in the frontier race — emollick · 2026-11-11
- Gemini Pro user's screenshot shows a locked '4 Carbon' model tier in the UI — SR_RSMITH · 2026-11-11
- xAI dumps JAX, rewrites full RL/inference stack in C/C++ as Grok 4.8 enters RL — ivan_bezdomny · 2026-11-11