Seeding checkpoints lifts deep-state success from 0% to 10.4%; StatePortal hits 27.5%

StatePortal: Agentic Checkpoint Synthesis for Testing Deep States in Interactive Software

Zhiyuan Peng, Wei Tao, Chenhao Ying, Yuan Luo, Yiwen Guo

Shanghai Jiao Tong University / LIGHTSPEED / Independent Researcher

ICLR 2027

cs.SE, cs.AI

2026-09-30

StatePortal synthesizes a checkpoint from a test case and read-only source. Seeding a pixel agent lifts deep-state success from 0% to 10.4%; hook execution reaches 27.5%.

What problem this solves

The behavior worth testing in an interactive program often sits behind a long chain of earlier actions. That is a deep state, and every commit pays to reach it again. Self-rendered engines, common in games, paint a single GPU surface and expose neither a DOM nor an accessibility tree, so selector scripts have nothing to hold. A pixel agent can click through the prefix, but each step is another model call, and an early miss spoils the rest of the path.

Open-source games barely automate this. Only 7.33% of 491 projects ship automated tests. In the appendix survey of 47 Unity, Godot, and Unreal repositories, no test plays a game into a state that interaction is required to reach.

Method

StatePortal takes a natural-language functional test and read-only source, and emits a checkpoint: a replayable package that restores the target state. Restoration goes through a hook, a callable state-transition interface onboarded once per application. The grader's specification stays hidden while the checkpoint is written.

An onboarding agent binds or writes that hook from a capability brief and runtime notes, with no hand-written hook code. All ten applications succeed: three through an existing API, three through an extension point, four through a test-gated entry. gpt-5.6-sol spends 5.49M tokens and 2,838 seconds. An independent rerun also succeeds, at about 6.07M tokens and $7.

Each test is one joint generation. A ReAct loop gets at most 16 turns, and one listdir, grep, or readfile per turn. A single final answer carries the restore strategy, payload, Start state, finite action list, and predicted assertions. Assertions are not scored. The payload is written into a live process, replacing the pixel prefix. The action under test runs only after state, process health, and, where the checkpoint claims it, a save/reload round trip have passed.

A trial passes only when a programmatic check and a visual check both pass. The visual judge, qwen3-vl-flash, may turn a programmatic pass into a failure and may not do the reverse. On 200 samples held out from the main study, two blinded models agree on 199 (κ = 0.9896). Against that consensus Qwen's strict accuracy is 96.0%, with a 3.39% false-accept rate and a 4.88% false-reject rate. Of 80 visual faults that leave every programmatic field unchanged, the hybrid oracle catches 77 (96.25%).

DeepStateBench is 61 tasks on 10 applications, five language families and six rendering stacks, from 874 lines to about 7.2M in LibreOffice. Games contribute 46 tasks. Selection weights were frozen before ranking, and all 61 references pass both qualification runs (122/122).

Results

The registered study is three models, seeds 0–2, and 1,647 trials, with the pixel action space held fixed.

SetupStartEnd-to-endMean tokens
Agent S3 from the initial screen7.8%0%408.3k
Agent S3 after a restored checkpoint36.6%10.4%163.8k
StatePortal, action via the hook38.4%27.5%106.7k

Task-clustered 95% intervals are [4.9, 16.6] for 10.4% and [19.3, 36.1] for 27.5%. Exact McNemar tests remain significant after Holm correction (p ≤ 0.00195). Start rates of the two seeded setups are close, so the 17.1-point end-to-end gap is the hook executing an action synthesized together with the checkpoint. Requests per trial fall from 42.4 to 16.2, and to 7.4 for StatePortal.

On gpt-5.6-terra the three end-to-end figures are 0% / 302.0k, 14.2% / 96.6k, and 24.6% / 51.6k tokens. StatePortal reaches 35.5% on gemini-3.5-flash and 22.4% on kimi-k2.6. gpt-6-astra, kept out of the registered pool, is still 0% from a cold start and 30.6% with StatePortal.

A separate planner that writes the action outside the exploration scores 24.2% end to end, below the joint 27.5%. It restores Start more often (43.9% vs 38.4%) but converts Start into Goal only 55.2% of the time, against 71.6% when the action is synthesized with the checkpoint.

Replays of 112 checkpoints, 1,361 runs and zero model calls, reproduce the original verdict 98.90% of the time, median 422 ms. On 11 merged fixes that every registered test passes on both the buggy and the fixed build, StatePortal separates the two rendered outcomes on 7 of 8 comparable tasks. Seeded pixel agents manage 4/8 and 2/8. Same-version pairs are wrongly separated 4 times out of 26 (15.4%).

Why it matters

One hook per application and one synthesis per deep state, and later regression runs call no model. Seeding a pixel agent only moves end-to-end success from 0% to 10.4%, so the absolute rate stays low. The use that holds up is differential testing: the same checkpoint places both builds in one deep state, and the rendered gap can expose a visual defect no checked-in test encodes.

Fifty of the 61 registered actions go through a source API, while the pixel baselines still click. Part of the 17-point gap is that interface.

Limitations

Start succeeds on 38.4% of trials and end-to-end success is 27.5%. Most tasks still fail. Eleven issues are a thin defect sample, and a 15.4% false-separation rate on same-version frames means a pixel diff will misfire. Seven of eight separations are too small a base for a regression gate. Across the 1,361 replays, one checkpoint is inconsistent and another is consistent but disagrees with its original verdict.

Visual calibration uses blank frames, striped occlusions, and wrong states. It does not sample real rendering bugs. A 4.88% false reject becomes a hard failure under the conjunction rule. Onboarding sees a capability brief and runtime notes. Four applications change state through a test-gated entry, so a pass is not evidence that a player clicked the same path.

Fan-in and fan-out are name-level text proxies, not a call graph. The paper does not repeat the study on open weights.

Terms

Source

Related papers

All paper explainers