HumanCLAW: Can Vision-Language Models Act Through a Body?
Siyao Li, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, Michael Zollhoefer, Sizhe An, Abhay Mittal, Amy Zhao, Ranjay Krishna, Manling Li, Ziwei Liu, Chuan Guo
cs.CV, cs.RO
2026-07-30
HumanCLAW has off-the-shelf VLMs drive a full body via atomic skills in a half-physics simulator; the best of nine frontier models (Gemini-3.1) reaches only 16.8% success.
Existing evaluations of vision-language models (VLMs) either keep them motionless (pure perception) or entangle decision-making with control in end-to-end training, so they never isolate the model's own action intelligence: whether it can decide, step by step, what a body should do next. This paper measures that ability alone, using a simulator to factor out the execution-side trouble (balance, motor error) and leaving only the "read the scene, decide the next move" part to the VLM.
HumanCLAW harnesses an off-the-shelf VLM: it takes egocentric observations and emits atomic skill commands, each realized by a Motion Base DiT (38M parameters) into a sub-second chunk of full-body motion with real physical consequences. The simulator is half-physics: it keeps collision, contact, and gravity but removes balance and motor failure, so failures attribute to decisions rather than execution. The skill set covers walk, side-step, back, turn, stairs, sit, and stop. The accompanying HumanCLAW-Bench has 1,218 egocentric full-body episodes across 41 indoor rooms, with the task "find a target object, bring the body to it, sit on it," stratified by distance, branching, and obstacles.
Nine frontier VLMs (GPT-5.5, Gemini-3.1, Claude-4.8, and others) were tested; none solves it. The leader, Gemini-3.1, reaches only 16.8% overall. On the three sub-metrics: FindSR ranges from InternVL3.5's low to Gemini-3.1's 64.9%; NavSR tops out at 42.4%; InteractSR peaks at 16.8%. Perception is essentially intact; once a target enters view it is almost always recognized. The breakdown is embodied self-awareness: models cannot tell whether they have arrived, where their body sits in the scene, or whether they have collided. The strongest open model, Gemma-4-31B, is close to the frontier (58.1/28.7/11.1).
It turns the abstract question "can VLMs act in a body?" into a quantified gap: perception is there, the bottleneck is self-localization and spatial judgment. For embodied-AI researchers it points at the next hole to fill; for model selection it warns that dropping a frontier VLM in as a robot's brain has a clear ceiling today. The authors consider training body awareness tractable, and this paper supplies the baseline.
The skill vocabulary is small, and manipulation beyond sitting is not covered. Half-physics deliberately drops balance and motor tracking, and there is no tactile channel, so collisions only displace the world and are never felt. The study tests only frozen off-the-shelf VLMs without a direct comparison to fitted action policies, so the claim that decision-making is the bottleneck is, by the authors' own label, a working hypothesis rather than a settled result.