IVG-Bench: closed-loop eval finds best VLM completes only 53.4% of interaction tasks
Chenhao Zheng · hf · 2026-11-12
University of Washington and Salesforce AI Research introduced Interactive Visual Gym (IVG), a framework for closed-loop evaluation and training of visual interaction models by jointly simulating task environments and human users:
- Three simulators cover cooking, household manipulation, and mobile-interface domains, with persona-conditioned user simulation injecting controlled execution errors and off-plan behaviors
- IVG-Bench instantiates 315 interaction scenarios; the best model, Gemini in Argon, completes only 53.4% of tasks within the time limit, showing weak factual grounding, unactionable guidance, and latency-driven failure to prevent errors
- A human study finds 78.5% agreement between the interaction-quality metric and participant preferences
- Proof-of-concept GRPO closed-loop training improves both task success and interaction quality for Qwen3.5-4B
More from Research
- Huawei releases GGUF weights for Qwen3.6-35B-A3B-MACaQ alongside paper and code — pmttyji · 2026-11-12
- E11 Bio hiring computational scientist for petabyte imaging and agentic AI labs — anderssandberg · 2026-11-12
- EPFL: Claude writes EEG network weights by reasoning, matching gradient training with 6x fewer subjects — epfl-ch · 2026-11-12
- Differentiable SSSS framework brings single-image real-time skin digitization to game engines — ssh4net · 2026-11-12
- NeurIPS paper proves when RNN+MLP can provably recover true robot state — chriswolfvision · 2026-11-12
- Curated 9-paper reading list for LLM quantization beginners, from QLoRA to BitNet — East-Muffin-6472 · 2026-11-12