AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark
Ruicong Liu, Liangyang Ouyang, Fengbo Lan, Zhixiang Wang, Kaipeng Zhang
cs.CV
2026-10-22
LoRA-tuning 33B MiniMax-H3 on 10,330 clips lifts game-entity temporal match from 50.9% to 62.9%, and missed count changes fall from 34.7% to 21.3%.
A coding agent can hand back a playable small game in minutes, logic and camera included. The frames are still programmer art: gray boxes, colored primitives, flat blobs. A studio would spend months modeling, texturing, and animating that away.
Models that generate the next frame from player actions, Genie and GameNGen among them, are better simulators than rule engines. Head counts and scripted events slip. A second line leaves state to the engine and generates only pixels. CWM, PWM, Matrix-Game 4.0, StreamRender-H3, and SolarRenderer sit in that line. Their public results cluster on third-person, first-person, or driving views, and several need class labels, state-carrying 3D boxes, or low-poly proxies. The games agents actually spit out are tower defense, auto-battlers, top-down survivors, turn-based battles, kart races, and crowd runners. What those pipelines have on hand is a crude video.
AnyGameRenderer consumes that video. It holds the camera, the layout, and every entity's position and entry/exit frames, then repaints the look from a prompt.
The backbone is MiniMax-H3 ref2va, a 33B model that edits a reference video from text. The crude clip is the reference. A six-part caption names the target style and the finished identity of each colored shape. An anchor image is optional. Training uses the first frame of the target clip; inference can swap in another style image or drop the anchor. Resolution tops out near 768p at 24 fps.
The frozen 33B weights stay fixed. Training updates a rank-128 LoRA, a low-rank adapter, on the self-attention and feed-forward layers of every transformer block. The loss is flow matching: fit a velocity field that carries noise to the finished frames. Every reported sample is 20-step Euler on the base plus this LoRA.
Four rules turn the caption into a correspondence contract. Shapes are stand-ins, the text names the finished character, and one color is one class. The retention segment demands that position, apparent size, timing, and head count be copied, and that empty regions stay empty. Wording stays affirmative. The base model has no negative prompt, and naming an unwanted HUD is a reliable way to get one. Captions are filled by code from metadata and detections, and they list characters that belong to the game but never enter the clip. That extra text is intentional noise, so presence is decided from the video.
The mix is 10,330 pairs, about 14.8 hours, all cut to 124 frames at 24 fps (5.17 seconds) and 1344×768. Crude and styled clips share a camera by construction.
| Source | Crude input | Clips | Share |
| GEAR, 38 agent games | primitive pass (coarse, semantic, and depth also stored) | 3,221 | 31% |
| Unreal two-stream | gray blockout with colored character blocks | 3,910 | 38% |
| Real gameplay | 3D boxes drawn as solids, wireframes, cylinders, or ellipsoids | 3,199 | 31% |
GEAR covers shooters, survivors, tower defense, karts, arenas, and co-op. Its training pair is the primitive pass against the full render. Unreal scenes are scripted around failures of the zero-shot model: same-color groups grow or shrink between 2 and 20 people, walking in or vanishing mid-clip; pixel height runs from 20 to 600; cameras are 35% follow, 30% top-down, 20% isometric, and 15% side; the protagonist is blue and visible on at least 95% of frames. Real footage is rebuilt from detected 3D boxes, in the spirit of PWM, across combat, driving, and pedestrians, with about 800 clips per proxy shape.
Training ran on 16 H200 GPUs with global batch 16. The learning rate warms up for 100 steps, then cosine-decays from a peak between 2×10⁻⁵ and 2.8×10⁻⁵, with weight decay 0.01. A step takes about 129 seconds, and three epochs take about 48 hours. The LoRA scale is 1.0 in domain. With no anchor and a style absent from training, the scale is set to 0.7.
The held-out benchmark has 59 cases and 753 entities: 34 from six GEAR games excluded from training, 11 Unreal blockouts, and 14 real recordings. The style or the cast is unseen. Entities that last under 12 frames are dropped. SAM3 does open-vocabulary segmentation from the caption's subject phrases, plus generic person and vehicle words: a text prompt finds the masks. Shadows and reflections are removed, masks are linked into tracks, and tracks are matched to input boxes with the Hungarian algorithm. A pair counts if IoU is at least 0.3, or the centroid shift is within half the input box. The same detector on ground-truth RGB produces the GT floor, a ceiling for what the pipeline can measure.
Temporal match is how much the frames of existence overlap the frames where the track is actually in the right place. Position match grades centroid error on those matched frames. Unrendered rate is the share of entity-frames that remain flat color. Hallucination rate is the share of detected entity-frames with no input entity. Across 75 count jumps, a rise of at least two entities inside a roughly six-frame window, the share that never catches up within the next 60 frames is the missed-count rate. On time means the onset error is at most 4 frames, about 0.17 seconds.
| Metric | Zero-shot | Fine-tuned | GT floor |
| Temporal match | 50.9% | 62.9% | 80.9% |
| Position match | 57.1% | 61.9% | 79.3% |
| Unrendered rate | 4.1% | 2.9% | 0.1% |
| Hallucination rate | 52.1% | 46.0% | 34.1% |
| On-time appearance | 55.9% | 67.6% | 93.6% |
| Never appears | 26.0% | 20.2% | 0.7% |
| Missed count changes | 34.7% | 21.3% | 0.0% |
Gains split by source. On Unreal, temporal match goes from 80.2% to 96.6%, close to the GT floor of 98.5%, and the never-appears rate falls to 0. On GEAR it goes from 47.6% to 60.7%, and missed count changes from 40.7% to 23.7%. Real recordings go from 46.5% to 51.1%; unrendered rate moves from 2.6% to 2.9%, and missed count changes stay at 20.0%. Temporal match beats the zero-shot base in all five genres, with the largest lifts on crowd and third-person games. Hallucination ticks up on top-down and third-person, and top-down remains the weakest. None of these numbers is a comparison against CWM, PWM, Matrix-Game 4.0, StreamRender-H3, or SolarRenderer.
A game pipeline can stop at code plus a blockout. Switching style is a prompt or an anchor change, so modeling, texturing, and animation can wait. One checkpoint covers many genres, and the game does not need to export a G-buffer. Roughly 10k pairs, 16 H200s, and 48 hours of LoRA is a smaller job than full-tuning 33B weights.
The measured gain is incremental. Temporal match rises 12 points and is still far from the 80.9% ceiling of the detector. A 46% hallucination rate means nearly half of detected entity-frames have nothing to match in the input. Missed count changes at 21.3% are too frequent to treat the output as the level's books.
Real recordings are the closest setting to restyling a shipped game from detected boxes, and they move the least. Missed count changes are flat, and the unrendered rate rises 0.3 points. The near-perfect Unreal numbers sit on scenes written to attack known zero-shot failures: same-color crowds, a six-color cast, and a blue protagonist on screen almost the entire clip.
The GT floor still shows a 34.1% hallucination rate, and temporal match saturates at 80.9%. SAM3 is not shown to hold up on non-human, heavily stylized characters, so painter errors and detector errors are tangled. Targets under 12 frames, including projectiles, never enter the score. There is no preference test. Clips last 5.17 seconds, and the latency of 20-step sampling on a 33B model is unreported.
At inference the crude video is not a sufficient input. The six-part caption has to name identities and lock counts and empty space in affirmative sentences. Unseen text-only styles also need the LoRA scale hand-set to 0.7. GEAR was trained as primitive-to-final pairs; semantic maps and wireframes do not get a per-input-type score.