DeepMind and Artificial Analysis Launch MKA Benchmark for Multimodal Agents

Google DeepMind partnered with Artificial Analysis to release the Multimodal Knowledge Agent (MKA) benchmark, which evaluates autonomous AI agents on complex multi-step visual knowledge tasks across professional domains. Claude Fable 5.2 topped the leaderboard at 57%, with Carbon second at roughly 1/6 the cost; the dataset is publicly available on Hugging Face.

Confirmed

• The benchmark was jointly released by Artificial Analysis and Google DeepMind, testing agents' ability to process complex visual documents and produce deliverables (slides, tables, reports, annotated images)
• It spans 10 professional domains with 200 tasks; test files include engineering schematics, architectural drawings, and clinical scans
• Claude Fable 5.2 leads at 57%; Carbon ranks second at 1/6 the cost
• One public example task is auditing architectural floor plans: checking for mismatched doors and windows between section and elevation drawings; Fable 5.2 viewed images 203 times to complete it, spending $68
• Per Artificial Analysis, a single task can run up to 224 turns and cost $68, underscoring the multi-step, high-cost nature of the tasks

Why it matters

• MKA targets visually complex tasks in everyday professional work that exceed pure-text capabilities (cross-checking drawings, reading scientific charts), filling a gap in multimodal agentic evaluation
• The cost gap among top models is striking: Carbon took second at far below the winner's cost, showing cost-effectiveness is increasingly critical in agent evaluations
• Tasks routinely requiring hundreds of turns and tens of dollars give a concrete sense of the real cost scale of deploying frontier multimodal agents

2026-11-11 ~ 2026-11-11 · 6 related posts

• Google DeepMind releases MKA benchmark testing agents on 10 professional visual work domains — _akhaliq · 2026-11-11
• DeepMind-backed MKA benchmark crowns Claude Fable 5.2 at 57%, with 30x cost spread — ArtificialAnlys · 2026-11-11
• Google DeepMind Launches MKA Benchmark: One Task Cost Claude $68 Across 224 Turns — ArtificialAnlys · 2026-11-11
• [source] Google and Artificial Analysis launch MKA benchmark; Gemini 4 Carbon ranks #2 at 6x lower cost — GoogleDeepMind · 2026-11-11
• [source] Inside an MKA task: auditing building plans costs Claude Fable 5.2 224 turns and ~$68 — ArtificialAnlys · 2026-11-11

1 near-duplicate retellings: ArtificialAnlys