MMDM Benchmark: 5,448 Questions Show Decision Models Cap Out at 36.8% on Hard Multi-select
CUHKDBGROUP · hf · 2026-11-11
Researchers from CUHK, HKUST and others released MMDM (Massive Multitask Decision Making), a benchmark for System-1 decision models that output structured probabilistic decisions in a single forward pass. It contains 5,448 questions across 16 task categories — agent state tracking, tool/skill selection, preference judgment, rubric scoring — with option spaces of 2–1,024 and inputs up to 64,817 tokens, built from fresh contexts to avoid test-set contamination. Evaluating 28 decision models shows strong world knowledge but weak joint reasoning: Perplexity Decider-V1.2 scores 78.75% on hard single-select world knowledge but only 53.95% on option logic and 47.95% on constraint planning, and the best hard multi-select accuracy across all models is just 36.81%. The authors flag joint reasoning, constraint-aware selection and complete multi-select recovery as key development priorities.
More from Models
- Claude Fable 5.2 Finally Solves Origami Task That Stumped Earlier Versions — FinanceYF5 · 2026-11-11
- Frontier AI gains steady +16.5 ECI/year, Anthropic's Fable 5.2 takes the lead — sebkrier · 2026-11-11
- If Google keeps this pace, it will be the best AI lab by year's end — imjustnewatai · 2026-11-11
- Fish Audio's new voice model panned: full-duplex claims don't hold up, hype is paid — realmrfakename · 2026-11-11
- Claude autonomously finds evidence against claim that mitochondria run 10-15°C hotter — paraschopra · 2026-11-11
- Auto Ontology benchmark: Google Surf MCP 116/120 edges out NVIDIA 113/120 — GarrixMrtin · 2026-11-11