8B UI Agent Trained With Online RL on Live iOS Apps Ties Gemini-3.1-Pro on UINavBench
Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps
Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio, Emmanuel Abbe
cs.AI
2026-10-29
Apple trains an 8B planner with online PPO on 512 concurrent iOS VMs; UINavBench success rises from 16.7% to 32.4%, tying Gemini-3.1-Pro, though iOSWorld stays low.
What problem this solves
Running a frontier model on a phone UI can cost several dollars per task, which rules out everyday use. Small models trained with SFT (imitating human demonstrations) tend to memorize trajectories, and one wrong tap leaves them with no idea how to recover. Existing online RL work mostly pays out only at the end of a task, so ten steps of good progress earn nothing if the last step fails.
This paper trains a small planner with online RL on real iOS apps and adds feedback for intermediate steps.
Method
Three pieces carry the design.
- Split planning from grounding. The Planning Agent (Qwen3-VL-8B, warm-started with SFT on human trajectories) reads the screenshot and history and outputs a step plan plus a remaining plan. A frozen internal 3B grounding model turns the plan into an action and coordinates. Grounding is close to a deterministic mapping that SFT handles well, so the expensive RL budget goes to the planner. Swipes are routed to Gemini-3.5-Flash because the 3B model only supports directional swipes.
- Two reward channels, both from LLM judges. Gemini-3.5-Flash scores every step as success, fail, or overcomplete (acting after the task is already done). Lucky progress from hallucinated reasoning counts as a failure, and repeated failed actions are penalized. Gemini-3.1-Pro judges the whole trajectory from the last three screenshots plus the action history, with a blocked-complete label (+0.5) for hitting an unavoidable barrier like a paywall or CAPTCHA.
- PPO with asymmetric discounting. Dense rewards use gamma = 0.5; the terminal reward is undiscounted. The dense sum is capped at 2.0, matching the +2.0 terminal success. With no discounting, an agent can pad the trajectory to farm per-step reward, which the authors saw happen.
They also propose GiGPPO. GiGPO (a GRPO variant) builds step-level baselines from identical recurring states, but with 4 to 8 rollouts per task on live devices, identical screens almost never repeat. GiGPPO instead finds the nearest sibling state by 64-bit perceptual-hash Hamming distance and feeds its return and the distance into a small MLP (about 0.9M parameters) on the critic, so the critic learns how far to trust the neighbor. Training also needs a 15-step critic warm-up with the policy frozen; otherwise advantages are dominated by value error.
Infrastructure uses cloud iOS VM snapshots (about 10 seconds to reset) and runs 512 trajectories at once. Each update samples 16 tasks with 8 rollouts each, capped at 30 steps per episode.
Results
| Setup | UINavBench | iOSWorld | AndroidWorld |
| SFT baseline | 16.7% | 4.5% | 4.7% |
| PPO (terminal only), step 336 | 21.6% | 7.5% | 19.8% |
| PPO, step 336 | 32.4% | 2.3% | 32.8% |
| GiGPPO, step 336 | 28.4% | 9.8% | 31.9% |
| Qwen3.8-27B | 19.6% | 19.5% | 49.1% |
| Gemini-3.5-Flash | 28.4% | 12.9% | 57.8% |
| Gemini-3.1-Pro | 32.4% | 39.5% | 60.8% |
On UINavBench (116 tasks on real iOS apps), PPO ties Gemini-3.1-Pro and beats Flash and Qwen3.8-27B. Trained only on iOS, the agent reaches 32.8% on AndroidWorld versus 4.7% for SFT, the roughly sevenfold gain in the abstract. Dense rewards mainly speed up learning: PPO at 168 steps (one epoch) already matches or beats terminal-only PPO at 336. On UINavBench, GiGPPO at 75 and 168 steps lands near PPO at 168 and 336.
Why it matters
For teams building cheap or on-device UI agents, this is a usable recipe: step-level LLM judging, discounting only the dense term, and a separately warmed-up critic. It also shows online RL on real devices can run at hundreds of parallel environments. The headline claim of matching Gemini-3.1-Pro holds on one benchmark only, so treat it as a local result.
Limitations
- iOSWorld barely moves. PPO at step 336 scores 2.3%, below the SFT baseline's 4.5%, and GiGPPO's 9.8% is far from Gemini-3.1-Pro's 39.5%. The authors blame synthetic app appearance plus personalization and multi-app tasks absent from training.
- On AndroidWorld the agent still trails frontier models by about 25 points.
- Benchmarks are small: about 116 tasks on UINavBench, so one task is roughly 0.9 points. No multi-seed runs or confidence intervals are reported. GiGPPO beats PPO on convergence speed but not on final UINavBench score (28.4% vs 32.4%).
- Evaluation is not pure 8B: grounding for iOSWorld and AndroidWorld uses Gemini-3.5-Flash, and training rewards depend entirely on Gemini judges, with no audit of judge bias.
- The gamma = 1 reward-hacking example comes from a GRPO run with a different, all-non-negative reward, so it is not a clean PPO ablation.
Terms
- GAE: generalized advantage estimation, which turns multi-step returns and a value estimate into a per-step advantage. Here lambda = 1, so it reduces to realized return minus the critic's estimate.
- Grounding: mapping an intent like "tap the search button" to screen coordinates.
- Critic: the value network in PPO that predicts future return from a state; discarded after training.
- Perceptual hash: a compact image fingerprint; smaller Hamming distance means more similar screens.
- Reward hacking: exploiting a flaw in the reward to score high without doing the task.
Source
Related papers
All paper explainers