Interactive Visual Gym: closed-loop eval and RL training of visual AI assistants with simulated users
jmin__cho · x · 2026-11-10
arXiv paper 2611.08286 introduces Interactive Visual Gym (IVG), the first closed-loop framework for evaluating and training visual interaction models, with authors including Silvio Savarese and Ranjay Krishna.
- Existing benchmarks replay prerecorded videos and can't measure how assistant advice changes user behavior and outcomes
- Three simulators recreate cooking, household manipulation, and mobile-interface tasks
- Persona-conditioned simulated users act, make mistakes, ask questions, and respond to guidance, with scheduled out-of-plan events
- 14 assistants evaluated across 315 scenarios; the best performer is Gemini-4-Argo
- RL training in simulation improves both task success and interaction quality
Paper, project page, and code are public.
More from Research
- Developer recreates Cartesia's blog experiment, visualizing model router chunk maps on images — davidwromero · 2026-11-10
- Aaronson-Ambainis Conjecture proved — by AI, Scott Aaronson announces — fortnow · 2026-11-10
- A hands-on guide to single-neuron steering in Qwen3.5-4B: hooks, doses and weird neurons — Tasty-Ad-3753 · 2026-11-10
- MuJoCo Playground 1.0 ships full sim-to-real workflows with SO101 and YAM robots supported out of the box — philfung · 2026-11-10
- Coding agents pick up and propagate vulnerabilities from the codebases they work in, new study finds — jyangballin · 2026-11-10
- ChatGPT-5.6's 16-hour autonomous run credited in shockingly simple proof of Crouzeix's conjecture — FrnkNlsn · 2026-11-10