Words to World: A Benchmark Testing LLMs on Closed-Loop Real-World Physical Control
Xinnan Dai · hf · 2026-10-15
Benchmarking LLMs in Interactive Physical Environments
Most benchmarks use static text, offline tool calls, or simulations. This one puts LLMs in a closed loop with real hardware — robotic arms, cameras, sensors — via a harness system.
Setup
- Models issue instructions, receive physical feedback, interpret state, and adapt their next action
- Tests grounded perception, action planning, tool selection, spatial reasoning, feedback-driven correction, and robustness to execution uncertainty
Findings
- Characterizes how well representative LLMs translate linguistic reasoning into physical actions
- Common failure modes: hallucinated state changes, poor recovery from unexpected outcomes, unstable long-horizon plans
Team spans Michigan State, Zhejiang University, Microsoft, Rochester, Wisconsin–Madison.
More from Embodied
- Kyra Interface raises from Afore Capital to build proactive smart glasses — himanshustwts · 2026-10-15
- Tesla set to unveil a 'flying car' tomorrow — ChrisGPT · 2026-10-15
- Concept render proposes turning Waymo cars into mobile co-working spaces with desks and WiFi — iScienceLuvr · 2026-10-15
- SpecWM: Spectral Alignment Boosts World Model Planning Success by up to 28 Points — randall_balestr · 2026-10-15
- Researcher pushes back on "robotics will soon be solved" claim from top lab insiders — adam_karvonen · 2026-10-15
- Atlanta apartment complex deploys robot security dogs that patrol at night and talk to residents — 4KTV · 2026-10-15