GTAGym turns GTA V missions into 150 auto-scored tasks; best model hits 70.9%, 18.9 pts below human

GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning

Jiajun Chen, Shicheng Fang, Jiahao Lu, Yuxin Wang, Xipeng Qiu

cs.CV, cs.AI, cs.CL

2026-11-09

GTAGym rebuilds GTA V missions into 150 engine-scored tasks; the best of six frontier LLMs reaches 70.9%, 18.9 points below human, plus a generator for RL environments.

What problem this solves

Getting agents to act in commercial AAA games stalls on scoring before it stalls on models. Sandbox platforms like Minecraft verify task outcomes programmatically but sit far from the complexity of a real open world. Projects that do play commercial titles, such as SIMA, Cradle, and Lumine, fall back on OCR or humans watching recordings to judge success. Broad benchmarks like GameVerse and GameHorizon-Bench score progress with VLM milestone matching or offline trajectory review, and their verifiable online tracks end up reimplemented in Minecraft. What is verifiable is not realistic; what is realistic is not verifiable. RL is out of reach either way, since training needs thousands of environments that hand out rewards automatically.

GTAGym (Fudan University and Shanghai Innovation Institute, the OpenMOSS team) puts all four pieces in one place: objectives taken from GTA V main-story missions, outcomes verified from engine state, parallel web-based evaluation, and a generator that mass-produces RL environments with reward signals.

Method

Tasks come from missions. The authors watched walkthrough videos and manually decomposed 75 main-story missions into subtasks, annotating each goal, interaction constraint, and entity behavior, then rebuilt them as 150 tasks: 103 online 3D tasks across nine categories (road, on-foot, aquatic, vertical, indoor, and aerial navigation, plus following, interception, and evasion) and 47 image-based shooting and sniping tasks built from raw GTA V frames. The 3D world is a Babylon.js reconstruction on top of the open-source GTASZ project rather than the shipped game, which is exactly what makes engine state fully readable. Scoring is binary with no partial credit: a walking arrival requires ground contact within 3 meters, and interception requires holding the target at or below 3 m/s for 1.5 seconds. An agent's own claim of completion counts for nothing.

Interaction runs through a web server exposing one screenshot-in, keyboard-and-mouse-out interface, with parallel evaluation. Actions are key presses with hold durations (one-second cap) plus mouse movement, scrolling, and clicks; the simulation pauses during inference. Two branches share the same tasks. GTAVBench hands the model a text instruction over a JSON protocol with a ten-pair history window. GTAVBench-AGI is closer to how a human plays: the objective appears only in the first screenshot, and the Codex harness compresses interaction history automatically.

For RL, each task family becomes a template that fixes the scene, interaction logic, and evaluator while exposing parameters such as start, destination, route length, turn count, movement mode, NPC speed, and pursuer count. Users supply ranges or option sets; the generator samples within feasibility constraints and emits task batches whose outcomes serve as sparse terminal rewards.

Results

Overall accuracy on the 150 tasks:

ModelGTAVBench (JSON)GTAVBench-AGI (Codex)
GPT-6-Astra66.470.9
GPT-6.1-Sol66.066.9
GLM-5.3-Flash38.941.8
DeepSeek-V4.1-Flash33.838.2
Kimi-K328.433.1
Qwen3.8-27B24.234.2
Human (3 players)89.8—

The two GPT models sit more than 25 points above the rest. Static navigation is near saturation, with both GPT models near-perfect on on-foot, indoor, and vertical tasks. Dynamic control is where things fall apart: aerial navigation is 0 for every model under the JSON protocol and peaks at 8.3% (Astra) under AGI; interception reaches 41.0% for Astra and close to zero for everyone else; evasion tops out at 25.0%. Even humans manage only 45.8% on aerial. Across 900 annotated trajectories (565 classifiable first errors), Calibration and Regulation account for 63.2%: wrong direction or magnitude on an input, or maneuvers started too early, too late, or never revised. Within ten decisions of a first error, Astra restores progress 65.7% of the time and Sol 49.3%, against 21.6% to 25.6% for the other four. Token efficiency also splits the field: under the JSON protocol DeepSeek emits about 26 times Astra's output tokens for roughly half the score.

For RL, the generator produced 50 on-foot navigation instances, and Qwen3.8-27B was trained with GRPO plus rank-32 LoRA on eight H200 GPUs, with binary terminal success plus up to 0.3 distance-progress credit on failed rollouts. Held-out on-foot accuracy rose from 54.2% to 72.9% and overall from 34.2% to 38.9%, with spillover to vertical (46.7% to 73.3%) and aquatic (57.6% to 69.7%) navigation. Road driving stayed at 3.3%, aerial and interception at 0%, and shooting slipped from 73.6% to 68.1%. On 100 native GTA V tasks, success went from 15% to 23%.

Two ablations are blunt. Cutting the action-duration cap from 1 s to 0.5 s lifted GPT-6.1-Sol from 62.1% to 66.3% on the 103 3D tasks, at the cost of doubled rounds. Removing the inference pause, so the world keeps moving while the model thinks, dropped the same model 22.3 points to 39.8%, with following, interception, and evasion all at zero. Inference latency alone is a bottleneck for dynamic interaction.

Why it matters

For benchmark builders, this is the first framework to combine real AAA mission objectives, engine-state scoring, parallel web execution, and RL environment generation, removing human judgment from the loop so the suite can run at scale. For training, the generator offers a budget path: LoRA RL on cheap generated walking tasks transfers to vertical, aquatic, and even native-game tasks on eight GPUs. The error analysis sharpens the target too: perception and planning are not the bottleneck, closed-loop control is. Against a human baseline of 89.8%, the 70.9% best score leaves headroom, but frontier models are already near ceiling on static tasks, so future separation will come from the dynamic categories.

Limitations

The 3D environment is a Babylon.js reconstruction, not the native engine; the authors did test transfer to real GTA V, but 23% post-training shows reconstruction scores do not carry over directly. RL gains are uneven: flying and interception did not move, and shooting regressed five points, a familiar forgetting pattern; the cross-category gains look more like general input-skill spillover than task-level generalization. The human baseline is three players with one attempt each. The 0.9 to 10 point lift from the AGI branch partly reflects the Codex harness's longer memory rather than model skill, and compression itself causes goal drift (a Kimi rooftop objective turned into a driving goal after compaction), so protocol choice moves rankings. Template-based generation limits variety, and open-ended long-horizon play remains out of scope.

Terms

Source

Related papers

All paper explainers