Meta's MMI Benchmark: Top Open Models Get Cross-Modal Outputs Right Only ~25% of the Time
_dieuwke_ · x · 2026-11-14
Meta Superintelligence Labs released the Modality Maturity Index (MMI): 893 human-written prompts mixing text, image, audio, video and documents in both inputs and expected outputs. Even frontier open models return the right modalities only 25% of the time. Top open models: Muse Spark Open (Meta) 15.55, Nemotron 3 Nano Omni (NVIDIA) 13.67, Qwen3-Omni (Alibaba) 13.54. Live leaderboard on Hugging Face.
More from Research
- Skeptic doubts Anthropic achieved large-factor bpb gains at fixed compute — teortaxesTex · 2026-11-14
- Quantizing training to MXFP8 with stochastic rounding: the accuracy-speed sweet spot — snowclipsed · 2026-11-14
- Ant Group's Mara Chain: refine prompts/harness/code, matches GEPA with far fewer rollouts — jiqizhixin · 2026-11-14
- Notion Ran ZeroEntropy's Reranker in Production Long Before the Acquisition — CShorten30 · 2026-11-14
- AI cracks Gödel's 1,009-page notebooks, uncovering hundreds of unknown theorems in days — stevenstrogatz · 2026-11-14
- Google's Gemini 4 multiagent workflows solve 10+ open math problems, incl. 50M-line Lean 4 proof — michael_galkin · 2026-11-14