RealityBench: simulated worlds test 26 VLMs on physical work documentation
arnabdotorg · x · 2026-10-14
Relantic released RealityBench, a Blender-simulated benchmark evaluating whether vision-language models can understand and document physical work (e.g., rooftop solar installation, vehicle repair) as seen through a worker's camera. It spans 11 modifiable environments and two tasks: acceptability (is a view valid proof of work?) and documentation (can the model infer task-relevant state?), with programmatic scorers validated against human raters.
Key findings from evaluating 26 VLMs: OpenAI's Astra, Anthropic's Fable, and Gemini Flash lead on documentation but remain weak at reading measurements and judging small parts and spatial relations. On-device models like Google's Gemma and Liquid's LFM reach 80% of top documentation scores. Simulation results matched real-photo performance on comparison questions. The benchmark is open for early access.
Related event: RealityBench tests VLMs' physical understanding in simulated 3D worlds(3 posts)→
More from Research
- DatologyAI launches Datology Labs for open collaborative data science research — pratyushmaini · 2026-10-15
- Prime Intellect experiments show ECHO beats pure RL in turn-capped, no-thinking robotics-like environments — DimitrisPapail · 2026-10-15
- Incoming Berkeley professor lists open questions: pre/post-training alignment, safe agents — canondetortugas · 2026-10-15
- Berkeley's QF3: off-policy flow RL with filtered Q-gradients trains humanoids from scratch — berkeley_ai · 2026-10-15
- VLMs May Turn Clearly Superhuman in Vision Within a Year or Two, Researcher Predicts — arnabdotorg · 2026-10-15
- Andrew Perrault: eval environments released only when scorers match human majority — arnabdotorg · 2026-10-15