New MDU Benchmark Tests 45 VLMs on Multilingual Doc Understanding — Best Score Just 69.0
kalyan_kpl · x · 2026-11-14
A new paper introduces MDU, a multilingual Document Understanding benchmark for vision-language models:
- 23,146 examples spanning 13 tasks across five languages: English, French, German, Italian, and Spanish
- Evaluates 45 state-of-the-art open-weight VLMs (1.7B to 235B parameters) plus one commercial system under the same protocol
Key insights:
- Far from saturated: the best average score is only 69.0, and just 8 of 45 systems reach 65. Notably, the two best results come from 27B models rather than the largest models evaluated.
- The language gap doesn't close with scale: the spread between the strongest and weakest languages persists as models grow.
More from Research
- NeurIPS 2025 best paper: diffusion models start memorizing at ~300 steps per image — Mastbubbles · 2026-11-14
- TestPrune: pruning regression tests to unblock AI-driven autonomous bug fixing — burkov · 2026-11-14
- Toddlers learn new verbs only from responsive partners; a call to rerun the test with AI — AnnaCiaunica · 2026-11-14
- Context beats model choice: Databricks benchmark shows 91% vs 58% for data agents — CautiousUse8597 · 2026-11-14
- MIT professor to showcase autonomous AI agent swarms for metamaterials discovery — ProfBuehlerMIT · 2026-11-14
- Walden talk: base models improving fast, mid-training data commoditizing — ZeYanjie · 2026-11-14