The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents
Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Wen Li
cs.CV
2026-08-06
GHD distills a next-screenshot teacher into a prefix-only student, firing only when the student errs and the teacher recovers the action. No inference overhead; beats GRPO on both benchmarks by up to 17 points.
GUI agents operate phones by reading screens and tapping. Because online training is slow, brittle, and prone to irreversible side effects, most agents are trained offline from previously collected successful trajectories. Standard practice slices each trajectory into prefix-action pairs: predict the action from the current screen and history, then discard the next screen. That discard is the leak. The reason an action is correct often appears only on the next screen.
The paper's example: to enable Soft Wrap in an editor the agent must click Edit or View, but nothing on the current screen points to those entries until the menu opens. Standard imitation supervises only the "click Edit" label, with no supervision for the reasoning and no evidence in the current screen to reason from. The agent may guess the action but cannot justify it, and on steps that need app-specific navigation knowledge it never learns. The authors call this future-dependence: actions are predicted from the past, but the evidence that justifies them lives in the future.
GHD (Gated Hindsight Distillation) lets a teacher see the next screenshot during training as privileged information, distills its hindsight-corrected reasoning into a prefix-only student, and drops the teacher, the future, and the gate at inference.
Three pieces matter.
A parameter-shared teacher and student. Teacher and student share weights; the teacher's only edge is the next screenshot appended to its input. For each response the student samples on the current screen, its tokens are appended to the teacher's context under teacher forcing, and the teacher emits a per-token distribution. Crucially both distributions condition on the same student prefix, so the teacher supplies dense corrections along the student's own rollout rather than decoding a fresh sequence.
Why the future helps. Answering "what should I do?" from the prior p(z|s) leaves the model uncertain and error-prone; answering "what must I have done?" from the posterior p(z|s,s') is nearly determined, because the outcome is visible. Reasoning drawn from that posterior is the missing supervision, and once written down it becomes an explicit target. App knowledge gets stated rather than left implicit in an action label.
The gate, distilling only what helps. A response is distilled only when the prefix-only student fails AND the hindsight teacher recovers the demonstrated action. The check reuses the same step verifier as RL (coordinate tolerance δ=20, action type and argument match), filtering ambiguous transitions and unreliable explanations out of the student's supervision.
The loss is GRPO plus a distillation term (weight λ=0.1) using a symmetric Jensen-Shannon divergence over the top-K (K=100) tokens. Dynamic sampling retries up to three times per batch to find a "student fails, teacher fixes" example, 2.69 attempts on average.
The controlled comparison (Table 3, three independent runs) pits SFT, GRPO, and GHD against each other from the same SFT init on AndroidWorld (AW) and AndroidLab (AL):
| Backbone | Bench | SFT | GRPO | GHD |
| Qwen2.5-VL-7B | AW Pass@1 | 46.55 | 47.13 | 52.73 |
| Qwen2.5-VL-7B | AL Pass@1 | 29.71 | 31.93 | 43.10 |
| Qwen3-VL-8B | AW Pass@1 | 59.05 | 61.35 | 66.47 |
| Qwen3-VL-8B | AL Pass@1 | 39.13 | 37.43 | 54.11 |
GHD's gain over GRPO is +5.6 and +11.2 on the 7B, +5.1 and +16.7 on the 8B. AndroidLab, the harder benchmark full of app-specific navigation, benefits most.
Against other systems (Table 1, each in its own setting) GHD takes the best average Pass@1 among open-data models at both scales (7B 47.9, 8B 60.3). It is not the overall leader: UI-Venus-1.5-8B, MAI-UI-8B, and MobileAgent-v3.5-8B score higher on AndroidWorld (73.7, 70.7, 71.6), but they are open-weight without open data and differ in model family, data, and algorithm. The table gauges overall competitiveness rather than isolating GHD.
The ablations pinpoint the source. Adding the gate alone gives +0.71 over GRPO, dynamic sampling brings it to +2.43, and conditioning the teacher on the next screenshot supplies the biggest jump (+3.17) to reach 52.73 (Table 2). Most of the gain comes from turning the future into token-level supervision, not from extra rollouts. Table 4 is blunter: giving the teacher the reference action or the reference reasoning barely moves the needle (-0.43, +1.29); only the next screenshot helps (+5.62). The future screen, not the gold answer, is what carries it.
For anyone building GUI or computer-use agents, this is a cheap, orthogonal training gain: no architecture change, zero inference overhead (deployment keeps only the prefix-only student), and it reuses the future frames that offline trajectories already throw away. It stacks on top of existing data-synthesis and RL methods rather than replacing them.
The broader lesson is diagnostic. The prefix-action decomposition that offline trajectory training takes for granted leaks information by construction. Any agent task trained from successful trajectories with steps whose correctness is only visible in hindsight can borrow the idea: use the future as training-time privilege, then drop it at deployment.
The paper has no dedicated limitations section; the authors concede points inline. Training runs only 200 optimization steps, a small RL budget; screenshots are downsized to 420x896 (about 3x fewer visual tokens) at the cost of a lower initial SFT score than the published OpenMobile checkpoint; and the claim that dynamic sampling adds no material training time rests on non-dedicated rented GPUs, which the authors call indicative rather than a controlled systems comparison.
Two things stand out on reading. First, GHD's win is concentrated on AndroidLab; on AndroidWorld the lift over GRPO is a modest five points, so the method pays off most on hard steps where the evidence is hidden in the future and adds little where the evidence is already on screen. Second, the gate thresholds (success reward τsucc=1.45, coordinate tolerance δ=20) are fixed hyperparameters with no sensitivity analysis, and need not be optimal in another task domain.