558K evaluations show vision-language models inversely scale on spatial reasoning and STEM
UniversityOfEdinburgh · hf · 2026-10-21
Researchers from the University of Edinburgh, Imperial College London, and MiniLM.AI present systematic evidence that vision-language models exhibit inverse scaling: larger models in the same family underperform smaller ones on specific subtasks.
- Scope: 19 models across 4 families, 8 multimodal benchmarks (9 splits), 29,386 questions per model, 558,334 model-question evaluations in total
- Key finding: per-question pairwise analysis identifies 375 subtask-model pairs spanning 3,432 unique queries where a smaller model answers correctly but a larger one fails
- Inverse scaling clusters in spatial reasoning and STEM question categories, and replicates across independently developed architectures; 13 subtask-dataset pairs show the pattern across multiple families
Implication: scaling gains are not monotonic across capabilities, and model choice should be task-specific.
More from Models
- Claude Max now includes $100/month in API credits, screenshot hints at Opus 5.5 with 1M context — michalmalewicz · 2026-10-21
- OpenRouter listing shows GPT-6 Sol outpacing larger GPT-6 Luna, sparking debate — tinteh · 2026-10-21
- Full video: Pichai on Argon's 25 trillion tokens in 72 hours now on YouTube — rohanpaul_ai · 2026-10-21
- Is that "open-source" AI actually open? A 4-level framework, three models scored — alex_verem · 2026-10-21
- InternScience releases Sage-A1.5, a 35B MoE agentic model fine-tuned from Qwen3.6-35B-A3B — Skyline34rGt · 2026-10-21
- 21 LLMs, 7,980 code reviews: higher reasoning effort often reduced precision — Electrical_Walrus537 · 2026-10-21