Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents
Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu
cs.HC, cs.CL
2026-10-30
HCU exposes code, accessibility, and GUI channels to computer-use agents and lets the model choose; it tops OSWorld 2.0 at 72.37% Partial and cuts input tokens by up to 57%.
Most computer-use agents work the way a person would: take a screenshot, find a control in the image, send mouse and keyboard events. That loop imposes a human-interaction prior on a frontier model. An input field may already report its name and current value through accessibility data, yet the agent still locates it by pixel coordinates. A dataset one line of code could process gets moved around through repeated clicks and drags. As benchmarks shift to long-horizon workflows (OSWorld 2.0), this loop taxes both success rate and cost.
Earlier hybrid systems took partial steps. ALE and WeaveBench expose screenshots as tools beside the agent's programmatic tools but barely touch accessibility. UFO2, Tactile, and CUA Driver do use it, each with fixed opinions about how. The Microsoft team's move is to lay out all three channels and let the model decide.
HCU places three channels in one agent context: the programmatic channel (the agent's own shell, code, and file tools), plus semantic (AX) and visual (GUI) desktop tools attached over MCP. All three act on the same computer. HCU adds no planner and no subagents, so the same interface plugs into Codex, Claude Code, Gemini CLI, and Grok Build.
The AX channel carries most of the design:
Channel switching has a hazard: an AX action can pop a dialog over coordinates from an older screenshot, and a GUI action can replace a control an AX reference points to. Action guards check before every input. The AX target guard resolves each reference against live elements; the GUI layout guard compares the live window layout with the one bound to the last screenshot. On mismatch, input is blocked and fresh evidence returned. A supplementary skill, distilled from recurring mistakes in development trajectories, explains channel capabilities, batching, and recovery; it contains no task solutions.
OSWorld 2.0 full set (108 tasks):
| System | Partial | Binary | Cost/task |
| StateAct (GPT-6 Astra, offline subset, unranked) | 72.60 | – | – |
| CUA Driver (GPT-6 Astra) | 68.82 | 31.5 | $10.05 |
| HCU (GPT-6 Astra) | 72.37 | 32.4 | $10.03 |
| HCU (Claude Opus 5) | 72.06 | 35.19 | $22.67 |
In the matched same-harness comparison, HCU beats CUA Driver by 3.55 Partial points with 29.81% fewer tool calls at essentially identical cost. With Claude Opus 5, HCU adds 3.75 Partial and 3.76 Binary over the official leaderboard result at 20.26% lower cost.
On OSWorld 1.0 (361 tasks, GPT-5.4), HCU scores 65.59 versus 62.30, with 29.80% fewer steps, 56.85% fewer input tokens, and 55.36% lower cost. The gains transfer to Claude Opus 5.5, Gemini 3.8 Flash, and Grok 4.7 (+2.94 to +12.90 points, 8–55% lower cost). Ablations attribute most of the gain to AX: removing it costs 2.79 points, while removing GUI leaves score within run-to-run variation and raises steps 12.45%. The skill cuts malformed-argument rejections from 17.37% to 1.36%. Cross-platform results hold: WeaveBench 53.25% vs 46.37%, Windows Agent Arena 80.86% vs 71.89%, at 46–51% lower cost.
This is an interface change with no training involved, built on capabilities desktops already ship, and the code is open. For teams building agents, the actionable finding is that adding an AX channel is the most cost-effective move available right now: the released version costs $0.40 per OSWorld 1.0 task against $0.98 for CUA Driver while scoring 7.66 points higher. The paper closes by citing Sutton's bitter lesson: locking the interface to how humans operate software caps what stronger models can do. The trajectories also show very different use of the same interface. GPT-5.4 sends 80% of desktop calls through AX; Claude sends 94% through GUI but batches more often (58% of its acting calls carry two or more actions). Interfaces should expose options rather than prescribe usage.
The paper concedes that AX coverage depends on what apps expose: custom-drawn content can have no accessible elements, and virtualized items may not exist until rendered. The pixel channel is not going away.
Several comparisons are not clean. OpenAI's official 72.60 for GPT-6 Astra comes from an internal subset and is unranked; the Gemini and Kimi numbers predate the 0808 task release with unspecified task counts. Gemini's +12.90 comes mostly from tasks where CUA Driver hit the 100-step limit, so a larger budget would shrink that gap. In the ablations, HCU without GUI actually scores 66.08 against 65.59 for the full interface; GUI earns its place through step reduction, not score. Claude Opus 5.5 spends 6% more time under HCU despite the gains. The macOS evaluation covers 28 self-selected UI-dependent tasks. And the GUI vocabulary experiment shows that interface conventions themselves bias results: GPT-5.4 drops 2.03 points when given Claude-style action names, so model-directed selection does not fully neutralize interface preferences.