InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants
Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu, Lemeng Wu, Hai Helen Li, Yiran Chen, Raghuraman Krishnamoorthi, Qi Qian
cs.HC, cs.CV
2026-10-25
The assistant only speaks and a simulated human acts. On 120 tasks an oracle passes 100%, Claude Fable 5.1 passes 41.7%, and guidance and planning reach 20% and 25%.
A wearable assistant does not move the body. It watches the first-person camera, the egocentric view, and talks. The person acts, and only then does the next frame change.
Recorded-video benchmarks are open-loop. The model answers a question or decides whether to speak, and the clip stays the same. In real help, the instruction changes the action, and the action changes the next view. Seeing, remembering, and phrasing have to work inside one multi-turn exchange. Closing that loop with real people is expensive and hard to repeat.
InteractiveWorld runs on Isaac Sim, with more than 8,000 objects and more than 50 scenes imported from BEHAVIOR-1K. The assistant shares the view and the dialogue and, each turn, may speak, stay silent, or stop. A second MLLM plays the human and turns free-form guidance into one of 36 action primitives that name only the verb and the object. A full-body character then navigates and grasps, and the simulator renders the next first-person frame. Unsafe actions are rejected before they run.
A task passes only when every required object state holds. Mean completion rate, MCR, is the fraction of those goals met in a task. InteractiveBench places 120 human-reviewed multi-turn tasks on this loop, in six groups: step-by-step guidance, scheduling under time and tool limits, describing a scene to someone who cannot see, a one-shot decision, a warning before a mistake, and finding an object from memory. On Accessibility tasks the human agent gets no vision.
The human in the main table is always Gemini 3.1 Pro. It may call every legal action, but it must follow the assistant rather than its own plan. Vision is capped at 32 frames a turn, four frames per action. Past that, the prompt keeps the latest eight frames and uniformly samples 24 older ones. With this same human, an oracle assistant passes 100% of tasks. With no assistance, the pass rate is 0%.
Across 12 frontier models, Claude Fable 5.1 is the best and passes 41.7%. Every other model is at or below 33.3%.
| Model | Pass | Guidance | Planning | Decision |
| Claude Fable 5.1 | 41.7% | 20% | 25% | 85% |
| Claude Opus 5 | 33.3% | 15% | 25% | 70% |
| Muse Spark 1.1 | 31.7% | 20% | 25% | 55% |
| GPT-5.6 Sol | 30.8% | 20% | 25% | 70% |
| GLM-5.3-Flash | 30.0% | 15% | 20% | 50% |
| DeepSeek-V4.1-Flash | 12.5% | 0% | 5% | 35% |
Overall pass rates not in the table: GPT-5.5 30.0%, Claude Opus 4.8 28.3%, Kimi K3 24.2%, Qwen3.8-Max 22.5%, Gemini 3.1 Pro 20.8%, GPT-5.6 Luna 20.0%. Fable's remaining categories are Accessibility 30%, Prevention 50%, and Finding 40%. Its MCR is 54.8%. Missing one state fails the whole task.
Decision items are often a single look, and the best pass rate is 85%. Guidance and Planning need continuous tracking and replanning. The best scores are 20% and 25%, and both drop as tasks get longer. Claude Opus 4.8 scores 0% on Guidance. The first error on a failed task is a misread or a forgotten state 70.4% to 85.7% of the time. For Gemini 3.1 Pro, 69.8% of first errors are visual. Kimi K3 fails more from memory than from perception, 39.2% versus 33.8%. One bad read steers the instructions that follow.
Average scores on nine capability checks do not separate the leaders. Fable and GLM-5.3-Flash sit at 59.2% and 58.2%, but communication is 79.6% versus 69.8%, and task pass rate differs by 11.7 points. Even Fable's first errors include 13.6% where the plan was never turned into a step the human could run. Proactivity is the weak column: 44.4% for Fable, 21.9% for DeepSeek-V4.1-Flash.
Extra steps are counted only on tasks the model already passed, as a multiple of the oracle path. Guidance and Accessibility commonly cost 2.35x to 3.66x. Fable, best on pass rate, costs 6.75x on Accessibility and 1.20x on Prevention. Kimi K3 matches the oracle on Finding at 1.00x and costs 3.66x on Guidance. A hidden memory field at inference time raises Prevention and Finding by 5 to 15 points, and lowers Planning by 5 points for two of the three models tested. Swapping the human model keeps the ranking: Fable stays first as assistant, GPT-5.6 stays last. All four assistants score highest when Gemini 3.1 Pro plays the human. Fable's mean MCR on the three long categories is then 47.03, and 40.33 when GPT-5.6 is the human. The main table uses the easier partner. Models also ask the person to look around or flip a package to read a label. The prompt does not require that.
Question answering on a fixed clip cannot show that a sentence sent the person the wrong way. Here the reward is the simulator's object state, so the same signal can train a model without another model as judge.
A one-look decision already reaches 85% for the best system. Staying with a person through a long physical task still tops out at 20% and 25%. A gap of about 10 points on the communication check shows up as 11.7 points fewer finished tasks. Extra actions are the user's time. The piece to pick up is the testbed. The code is public, and running it means Isaac Sim.
The person in the main table is Gemini 3.1 Pro. Assistant rank is mostly stable across four human models, but Fable's MCR on the three long categories moves by about seven points. The paper leaves calibration to real users as later work. An appendix entry covers a check with real participants, and the main text does not report those numbers, so 41.7% is not a glasses-on-a-person result.
The human is told to obey and not to improvise. The 0% score with no assistance mixes task difficulty with that rule. A real user will drop a bad instruction and will also act early. The motor layer repairs small collisions and changes stance. Pouring and cutting follow scripted paths. An awkward line that the executor can still complete counts as success. The benchmark grades seeing, remembering, and speaking.
Efficiency ignores failed episodes. The 32-frame cap forces long tasks onto subsampled frames, and the paper does not split memory loss from dropped frames. Category pass rates move in steps of 5 points, so one task is a full bin. Of 51 scene layouts checked in the appendix, 44 admit generic tasks. The assets are still BEHAVIOR-1K simulation. The paper calls this the first environment that is at once egocentric, performed by a human, grounded from free language, conversational, closed-loop, and scored from state. Open-loop sets such as EgoSchema and HoloAssist never let the answer change the next frame. That gap is real. How the six boxes are drawn is still arguable.