MintAct: A Unified Visual Agent for Digital Environments
Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy, Kaixin Ma, Vishwesh Kirthivasan, Oğuzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan
cs.CV
2026-09-19
MintAct unifies grounding, mobile/desktop/web navigation, and visual tool use at 2B-8B. MintAct-8B reaches 48.9 on OSWorld-Verified and 67.0 on AndroidWorld in one checkpoint.
A pixel-level digital assistant has to ground instructions to screen coordinates, navigate multi-step tasks on mobile, desktop, and web, and call external tools from what it sees. Those capabilities are still trained as specialists. Observation spaces, actions, data, and runtimes differ, so naively merging action sets or mixing data lets domains interfere. Compact models meant for on-device use cannot afford one specialist per domain.
The hard part is not another VLM. GUI backends are slow and flaky, trajectories are long and multimodal, and asynchronous RL will drift toward whichever domain finishes rollouts first. Unification is a data, environment, and training-protocol problem.
MintAct is a 2B/4B/8B family from Apple, initialized from Qwen3-VL-Instruct. Every UI domain sees raw screenshots with coordinates normalized to 999×999, no DOM or accessibility tree. Action vocabularies stay separate; a domain system prompt steers the model into the matching tokens. Supervised and RL mixtures are balanced across domains on purpose.
Training has four stages. High-resolution single-step SFT builds grounding. Low-resolution multi-step SFT trains on environment trajectories, 25% per domain, capped at 30 turns (100 for tools). Per-domain RL specialists are then distilled back into one set of weights via rejection-sampled RFT. Final joint asynchronous RL runs only on mobile and desktop.
Desktop sits on OSWorld with more than 200 concurrent instances; mobile on AndroidWorld with more than 100. Web uses Weblica cache replay and synthetic sites. Visual tool use uses MM-ToolSandBox with a dynamic search over 500-plus native tools. The async stack decouples rollout from training, bounds the queue, syncs weights on a schedule, masks environment-failure trajectories, and applies truncated importance weights for engine-versus-trainer numerics.
MintAct-8B versus Qwen3-VL-8B: AndroidWorld 47.6 to 67.0, OSWorld-Verified 33.9 to 48.9, Weblica 55.5 to 74.7, MM-ToolSandBox 3.1 to 24.5. Among size-matched public models it leads on OSWorld-Verified (48.9 vs EvoCUA-8B 46.1), Weblica (74.7 vs WEBLICA-8B 70.6), UI-Vision (56.6), and OSWorld-G (64.5). AndroidWorld at 67.0 still trails MAI-UI-8B at 70.7. Unification does not win every cell.
| Model | OSWorld-Verified | AndroidWorld | Weblica | MM-ToolSandBox |
| Qwen3-VL-8B | 33.9 | 47.6 | 55.5 | 3.1 |
| MintAct-8B | 48.9 | 67.0 | 74.7 | 24.5 |
| Specialist | EvoCUA 46.1 | MAI-UI 70.7 | WEBLICA 70.6 | — |
Ablations split the recipe. High-resolution grounding alone drops OSWorld-Verified to 9.4. Multi-step SFT alone drops UI-Vision from 59.3 back to 25.0. Both stages are required. After RFT, joint RL pushes OSWorld-Verified from 43.1 to 48.9 and lifts web (Weblica 72.8 to 74.7) with a 1.1-point AndroidWorld dip. Synthetic-environment RL with no real emulators still reaches 60.3 on AndroidWorld and 38.6 on OSWorld-Verified; when real environments exist, real joint RL is stronger.
One 8B checkpoint can sit next to size-matched specialists on grounding, three-platform navigation, and visual tool use. That is the serving argument for a single on-device assistant. The more reusable piece is the async RL stack that holds a cross-domain mix instead of letting the fastest backend own the gradient. Synthetic environments transfer when real data is scarce and step aside once real rollouts exist, which is a practical budget order.
Joint RL still covers only mobile and desktop. Web and tool-use scores at the end of training owe more to SFT/RFT and transfer than to four-domain online optimization.
The authors list the main gaps. Joint RL was not extended to web and tool use because keeping every backend live is expensive. Every screenshot is appended, so long horizons blow up context. Visual tool use is still a parallel skill, not a policy that switches between pixels and APIs on demand. A dynamic tool registry rewrites the system prefix and hurts prefix caching. Navigation is capped at 30 steps and tool use at 100; longer tasks are unreported. The AndroidWorld gap versus MAI-UI-8B remains.