RealityBench: 35 VLMs tested on physical work understanding, measuring stays a weak spot
arnabdotorg · x · 2026-10-28
Relantic released RealityBench, a Blender-based benchmark across 11 simulated environments (rooftop solar, vehicle repair) evaluating how well VLMs document and understand physical work, with programmatic scorers, an arXiv paper and HF datasets.
- Across 35 VLMs: OpenAI's Astra, Anthropic's Fable and Gemini Flash lead on documentation, but models remain weak at reading measurements and judging small parts and spatial relations.
- On-device models like Gemma and Liquid's LFM reach 80% of top documentation scores.
- Simulation results match real photographs, suggesting sims predict reality.
- The authors note OpenAI's new Terra release performs surprisingly well, and that adding Clef reveals a new Pareto frontier: decisions are a cost-effective path to physical world understanding.
More from Models
- Early Take on Gemini 4 Carbon: Lots of Thinking, Good Model Overall — SamuelAlbanie · 2026-10-29
- Gemini 4 Carbon tops AA Index v5, beats Opus 5.5 and GPT-6 Astra on most benchmarks — Yuchenj_UW · 2026-10-29
- Gemini 4 Carbon Keeps $2/$10 Pricing, Scores 59 on AA v5.0 — _philschmid · 2026-10-29
- Reddit thread: picking third-party LLM inference providers, GLM Flash cited as best value — inwardPersecution · 2026-10-29
- Reddit reposts Gemini 4 Carbon announcement with benchmark table — Marimo188 · 2026-10-29
- Google Announces Gemini 4 Carbon, Its Next Frontier Model — patloeber · 2026-10-29