Scale AI's MA-Bench tests LLMs on military-style mission analysis — the best model scores just 62%
Rodney Lafuente-Mercado · hf · 2026-10-15
Scale AI and Data Machines introduce MA-Bench, which tests LLMs on mission analysis — the first step of military operational planning: reading an ambiguous, contradictory document set and stating the problem before proposing solutions.
- 10 expert-built scenarios pair 10–12 synthetic documents with 1,257 expert-written pass/fail requirements, each tagged with source, question type, and importance.
- A reasoning-model judge grades like a fifth credentialed planner without shifting the four-planner panel's agreement (Fleiss κ 0.42→0.41); a second provider's judge reproduces the ranking (Spearman 0.94).
- The best of 20 frontier and open-weights models gets just 62% of rubric items right; models are strongest on stated facts (58%), weakest on inferred judgments (33%), and 6 of 10 failures are omissions rather than errors.
- Packets, rubrics, prompts, scorer, and all graded briefs are released openly.
More from Research
- How did NLP researchers cope when LLMs solved their field? A lesson for what's next — iScienceLuvr · 2026-10-15
- Alibaba's HICO-Bench shows 36% of highly-rated agent conversations fail verification — Tongyi-ConvAI · 2026-10-15
- PixText fixes text in generated images: 83% glyph pass rate vs 37% for baselines — ATH-MaaS · 2026-10-15
- TextVerifier catches flawed Chinese glyph rendering with F1 0.844, far above OCR and VLMs — ATH-MaaS · 2026-10-15
- ByteDance's Diff Diffusion predicts code edit paths: 2.78x faster inference, 39% lower latency — Xiaolong Xu · 2026-10-15
- MimicX refines humanoid tracking with policy-in-the-loop feedback: 25.7% error cut — Shuaijun Liu · 2026-10-15