Salesforce's Interactive Visual Gym: best VLM finishes only 53.4% of 315 closed-loop tasks
jcniebles · x · 2026-11-10
Salesforce AI Research and UW introduce Interactive Visual Gym (IVG), an open simulation framework for closed-loop evaluation and training of visual AI assistants. Key points: simulators for cooking, household, and phone tasks with persona-conditioned simulated users who make mistakes and respond to guidance; IVG-Bench has 315 scenarios across 14 assistants — the best model, Gemini-4-Argon, completes only 53.4% of tasks on time, weak in factual grounding and often too slow to prevent errors; simulator scores match human preferences 78.5–80%; simulator training lifts a 4B open model's success rate from 33.3% to 42.2%.
More from Models
- System card confirms Claude MyTHOS 5 and Fable 5 share the same underlying weights — repligate · 2026-11-10
- Gr00t 2 is here: Jim Fan's talk teases NVIDIA's next-gen humanoid robot foundation model — zhengyiluo · 2026-11-10
- System card asserts with specificity that MyTHOS 5 and Fable 5 share weights — repligate · 2026-11-10
- Early Tester: Fable 5.2 Built a 3,100-City Water Map in One Go, but Still Makes Silly Errors — alliekmiller · 2026-11-10
- Fable 5.2 burns so many tokens its 20% price cut is effectively a wash — addyosmani · 2026-11-10
- Opus formalizes CPython in 613K lines of Lean — and finds real Python bugs — pbaylies · 2026-11-10