Agents hunting bugs in 3D worlds: best model finds 42%, humans 83% on WorldAuditBench

WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents

Ziyan Jiang, Jingbo Yang, Jiabao Ji, Yujian Liu, Qiucheng Wu, Tommi Jaakkola, Yang Zhang, Shiyu Chang

cs.AI

2026-10-01

WorldAuditBench asks multimodal agents to find planted bugs across 213 interactive 3D-world tasks; the best VLM agent scores 42.3%, two-stage VLA-VLM setups 17.4%, humans 83.4%.

What problem this solves

Interactive 3D worlds built on game engines are the main substrate for embodied AI research, and a world can render beautifully while still being broken: a solid wall you can walk through, a chair hovering off the floor, a sign that vanishes when you look away. Benchmarks run in buggy environments produce unreliable conclusions; a navigation agent that finishes by clipping through a fence has not learned navigation. Prior benchmarks miss this problem from both sides. GlitchBench and VideoGameQA-Bench hand the model pre-recorded footage and test passive recognition; EmbodiedBench and OpenEQA test task completion without asking whether the environment itself is sound. The missing piece is world auditing: an agent entering a world, gathering evidence, and confirming or rejecting suspected defects. Nobody had benchmarked that.

The hard part is that two capabilities must stay coupled: visual reasoning decides what to check, and action fetches the evidence. A fence that looks solid only reveals a missing-collision bug when the agent walks straight into it for 3 meters and compares views before and after. Observation raises the hypothesis; action tests it.

Method

WorldAuditBench contains 213 hand-crafted single-anomaly tasks across 13 environments: 126 on Unreal Engine 5 and 87 on Three.js, spanning indoor, urban, historical, industrial, and natural settings (27 scenes total). Anomalies fall into five families subdivided into fifteen categories:

• Static physics: floating objects, intersections, wrong scale; visible in a single frame
• Interactive physics: missing collision, invisible obstructions, abnormal contact responses; exposed only through action
• Spatial consistency: view-dependent appearance, detached shadows, occlusion errors
• Temporal consistency: objects vanishing or relocating between visits, abrupt state changes
• Semantic consistency: configurations that violate an object's function or the scene's era, like cola bottles in an ancient Chinese teahouse

Each task ships a scene description, an anomaly-type hint, and one in-context example; the target object, its location, and the grading rubric are withheld. Reviewers who did not build the tasks validated them in a browser, with acceptance requiring at least two Pass ratings on the same version.

Two paradigms are compared. A VLM-only agent works through an MCP tool interface with a 40-action budget, plus memory tools (retrieve earlier frames, take notes) and reporting tools (file a suspicion, revise it with more evidence). A VLA–VLM pipeline sends Open-P2P 1.2B to explore for 60 recorded seconds, then a VLM analyzes the fixed trajectory offline, closer to standard video QA. Both submit reports graded by a GPT-6 Astra judge against each task's rubric; judge-human agreement is 91.7%.

Results

• Across five frontier models and both paradigms, success runs 6.6% to 42.3%; humans reach 83.4%, roughly double the best model.
• Interactive auditing wins across the board: VLM agents score 28.2–42.3%, two-stage VLA–VLM auditors 6.6–17.4%. The first gap is exploration itself. Geometric coverage is 91.1% for the VLM agent versus 47.4% for the VLA explorer; human-rated exposure is 68.1% versus 34.7%. The 1.2B VLA often never reaches the anomaly in large worlds.
• The advantage survives on matched evidence: on tasks both paradigms covered, the VLM agent still leads 38.5% to 26.0% (62.1% vs 47.3% under human-rated exposure), with memory tools for retrieving earlier frames doing real work.
• Temporal consistency is where everything fails: 12.5% at best for VLM auditors, mostly 0 for VLA–VLM.
• Guidance ablations (Gemini, 126 Unreal tasks) price the hints: removing the in-context example costs 2.4 points, and removing the anomaly-type hint on top of that collapses success from 33.3% to 15.9%. Knowing what to look for beats seeing a demonstration.
• With 2–3 coexisting anomalies, anchor-anomaly success holds (42.9% to 47.6%), but finding every anomaly drops to 7.1% and 4.8%.
• VLA–VLM runs 2.7–10.3× faster per task at several-fold lower cost, a clear speed-for-recall trade.

Why it matters

For labs running embodied benchmarks, this is an automated testbed for the environments themselves, since environment bugs contaminate every downstream number; the bench turns 'checking the world' into a measurable task. For agent builders, two design conclusions come with numbers attached: reasoning-guided online exploration beats record-first analyze-later, and a retrievable evidence memory is load-bearing for cross-viewpoint and cross-time anomalies. Game QA automation is the most direct application, and the rubric-plus-judge protocol can be lifted wholesale. To be clear about what this is: a benchmark paper. No new method; every number comes from existing models wrapped in tooling.

Limitations

The paper states itself that the two paradigms use different budgets (40 actions vs 60 seconds), so the comparison is not perfectly matched. VLA exploration is represented by a single 1.2B behavior-cloning model, which says little about the ceiling of larger VLAs. Temporal-consistency tasks defeat every model while humans still score 78.1% on them, so the gap is real but measured coarsely. A few more caveats from close reading: 213 single-anomaly tasks is a modest scale; the judge is one model with moderate human agreement (κ = 0.645); each score reflects a model plus its CLI harness, so harness quality is baked into the numbers; and the human baseline, roughly ten CS PhD students at ten minutes per task, is not strictly comparable to a 40-action agent budget.

Terms

• VLA (vision-language-action model): a policy model mapping images and instructions straight to actions, common in robotics and game playing
• VLM-as-judge: grading agent reports with a multimodal model against a rubric instead of human raters
• MCP (Model Context Protocol): a standard interface exposing tools such as move, retrieve-frame, and file-report to a model
• Geometric coverage: a mechanical check on agent position and gaze, used as a proxy for whether the anomaly was ever in view
• Rubric: the human-written per-task spec of the target anomaly and its expected normal behavior

Source

• arXiv 2609.40325v1
• PDF
• Hugging Face
• GitHub · https://github.com/UCSB-NLP-Chang/WorldAuditBench

Related papers

• Early-fault success goes from 0% to 100%; late-fault tokens fall from 7566.7 to 3694.4
• Self-built ARC harnesses score from 0 to 97.14; two revisions add 2.80
• ToM2Vec adds 12 Overcooked points by aligning 10 latent tokens to a partner
• T2Quant reallocates a fixed 4-bit DLM budget over denoising steps, gaining up to 6.7 points
• 8-Hour CFD Cut to 3 Seconds: Neural Operator Wind Fields Save UAV Energy up to 12.7% in Headwind
• MongoDB and Trane propose contestable governance: agents argue, a steward contests before commit

All paper explainers