DeepMind's MKA benchmark splits frontier agents: 95-97% on basics, 28-57% on hard tasks
ArtificialAnlys · x · 2026-11-11
- Artificial Analysis launched its implementation of Google DeepMind's Multimodal Knowledge Agent (MKA) benchmark, testing whether agents can read visually dense professional files — architectural drawings, engineering schematics, clinical scans — and produce real deliverables (reports, spreadsheets, slide decks, marked-up drawings).
- Setup: 200 tasks across 10 industry verticals; agents run in a Linux sandbox with code execution, web search, and image viewing, capped at 250 turns. GPT-6 Luna (medium) serves as agentic judge, scoring weighted satisfaction rate (WSR).
- Scoring design: each rubric criterion is labeled Climb (hard, frontier models still fail) or Regression (baseline competence). The top seven models cluster at 95-97% on Regression, while Climb separates them across 28-57% — making it the real discriminator.
Related event: DeepMind's MKA benchmark exposes big gaps among top models(4 posts)→
More from Models
- TheZmi flags Fable 5.2 regression: actively less robust across model card benchmarks — TheZvi · 2026-11-11
- Siri AI's Skimmer: regex-only fast path calls tools up to 20× faster than LLMs — SumitGup · 2026-11-11
- Stripe CEO: Higher Reasoning Effort Thinks Harder Per Option, Not About More Options — gabriel1 · 2026-11-11
- Poll: 70% say Anthropic's best internal model plus $50M can dominate any human expert — JacquesThibs · 2026-11-11
- Ollama cloud 'DeepSeek' claimed it was actually ChatGPT, user alleges model swapping — No_Literature_7653 · 2026-11-11
- Rumor: Anthropic's Claude 4 said to be 'very different' in cyber and RSI capabilities — ChrisGPT · 2026-11-11