RealityBench: Blender-simulated 3D worlds benchmark 26 VLMs on physical work understanding
arnabdotorg · x · 2026-10-15
Relantic launched RealityBench, a portfolio of 11 Blender-simulated 3D worlds (rooftop solar installation and vehicle repair) for evaluating how well vision-language models document human physical work.
- Extendable, not a fixed image set: worlds are simulators with programmatic scorers (validated against human raters); reality can be "bent" to generate new ground-truth data.
- Two tasks: acceptability (is a view suitable proof of work) and documentation (understanding task-relevant state from a view).
- 26 frontier VLMs evaluated: OpenAI's Astra, Anthropic's Fable and Gemini Flash lead on documentation but remain weak at reading measurements and judging small parts and spatial relations; on-device models like Gemma and Liquid's LFM reach 80% of top scores.
- Early access is live on Hugging Face.
Related event: RealityBench Tests VLMs' Physical Understanding via Blender 3D Simulations(4 posts)→
More from Research
- Anthropic: model activations predict ripple effects of value alignment training across 66 values — dhadfieldmenell · 2026-10-15
- Three CMU researchers named 2026 AI2050 Early Career Fellows in $16M Schmidt Sciences program — AkariAsai · 2026-10-15
- Google's VEPA: lifelong formal guardrails that evolve with agent deployment — mohitban47 · 2026-10-15
- Study: the best optimizer depends on batch size, rankings reverse even after retuning — ChengleiSi · 2026-10-15
- Spectral's AI agent passes SOLIDWORKS Professional CAD exam in under 3 hours — katieruthmishra · 2026-10-15
- OpenAI's Navier-Stokes proof appears to use a fundamentally different approach, review finds — FlorianGallwitz · 2026-10-15