HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents
Tongbo Chen, Junbo Niu, Zhengxi Lu, Niu Lian, Fei Tang, Yuchen Yan, Yike Hong, Yong Du, Yizhou Liu, Bofan Chen, Yongliang Shen
cs.CV
2026-09-30
Naive CLI access drops Qwen3.5-9B from 38.8% to 18.4% on OSWorld. After hybrid-trajectory SFT plus CLI-aware RL, HybridCUA-9B reaches 53.6% (+14.8) and 14.0 average steps.
Computer-use agents (CUAs) still pick from two bad options. GUI-only policies are general: screenshots, clicks, and keystrokes work in any window. They also fail in chains. One missed click poisons the rest of a long trajectory. Application-specific APIs and MCP tools compress those clicks, but each app needs its own integration, so the speedup does not travel.
The command line looks like a third path. A single sed or a python-docx script can replace a dozen menu clicks, and the shell already ships with the OS. Off-the-shelf models do not know how to use it. On OSWorld, four representative agents drop 2.5 to 11.5 percentage points once CLI is exposed. Qwen3.5-9B falls from 38.8% GUI-only to 18.4%. Larger models barely route there: Qwen3.5-27B sends 15.0% of steps through CLI, EvoCUA-32B sends 0.15%.
Two training gaps sit behind this. CUA corpora are almost all GUI; terminal and code corpora have no screenshot context. Step-level imitation scores local plausibility, outcome rewards score task completion. Neither tells a well-chosen interface from a long, lucky GUI path.
HybridCUA comes from Zhejiang University, Peking University, and Tsinghua. The base is Qwen3.5-9B. Every executable act goes through one bash(command, timeout) call. GUI is pyautogui inside a quoted Python heredoc; CLI is a raw shell command. Wrapping clicks in bash does not reclassify them: only a command that actually invokes a shell program counts as CLI. The observation is the current screenshot, plus stdout/stderr when the last step was CLI.
A pipeline builds three trajectory types, collected as HybridCUA-8K:
The SFT split has 5,023 trajectories across 11 apps. Separately, 3,000 RLVR tasks get executable verifiers. For each task, 16 rollouts are sampled in GUI-only, CLI-only, and GUI-CLI; modes are ranked by success rate, with near-ties (within 5 points) broken by median successful steps. The label b-star is 1 when CLI has a clear execution advantage. All tasks are kept.
SFT runs 2 epochs on 46,876 step samples. Then GRPO on 1,000 tasks drawn from the 3,000. A binary success reward Racc treats a well-routed rollout the same as a long GUI slog, and does not punish a broken CLI command if the task later succeeds. Two CLI-aware terms fill that hole. Task-level RCLI is 1 only if the rollout succeeds and its CLI-or-not bit matches b-star (weight 0.1). Step-level rexec is -1 on a shell-level failure at a CLI step (weight 0.3), added to that action's token advantage.
OSWorld uses all 361 tasks, 50 steps max, one run per number. Accuracy is the mean verifier score; some verifiers return fractions, so the figure is an upper bound on strict success rate.
| Method | Action space | Acc. | Avg. steps |
| Qwen3.5-9B | GUI | 38.8% | 31.6 |
| same, CLI added, no training | GUI+CLI | 18.4% | 22.1 |
| +SFT | GUI+CLI | 46.0% | 19.8 |
| HybridCUA-9B | GUI+CLI | 53.6% | 14.0 |
| same base, GUI-only SFT+RL | GUI | 50.4% | 22.1 |
| AutoGLM-OS-9B | GUI+API | 48.9% | - |
| ToolCUA-8B | GUI+API | 46.8% | 14.9 |
| UltraCUA-32B | GUI+API | 43.7% | - |
| EvoCUA-32B | GUI | 56.7% | 25.0 |
HybridCUA-9B is 14.8 points above its base and 17.6 steps shorter, best on both metrics among similarly sized models. It leads AutoGLM-OS-9B by 4.7 points, ToolCUA-8B by 6.8, and 32B UltraCUA by 9.9. EvoCUA-32B still sits higher at 56.7%, as a 32B GUI-only agent.
Shorter traces after naive CLI access are early failures, not efficiency. At both training stages the hybrid action space beats a matched GUI-only recipe: 46.0% vs 44.2% after SFT, 53.6% vs 50.4% after RL. Mixing all three SFT sources hits 46.0%. GUI-only SFT is 43.2%, interleaved-only 41.0%, CLI-only 31.7% (below the untrained base).
RL lifts accuracy 7.6 points (46.0% to 53.6%) and cuts steps 29.3% (19.8 to 14.0). Dropping RCLI barely moves accuracy, but CLI share stalls at 58.9% versus 64.0%, and step reduction is 18.2% instead of 29.3%. Dropping rexec lets the CLI error rate climb back to 16.5% by training step 120, above the SFT level; the full model settles at 11.5%.
On OSWorld-MCP (same 361 tasks, MCP environment) HybridCUA-9B scores 47.1% against 38.0% for the base and 46.8% for ToolCUA-8B. On WindowsAgentArena (154 tasks) it scores 36.0% against 32.0% and 33.8%. It emits PowerShell despite training only on Linux shells. OS tasks send 84% of steps to CLI; Chrome tasks send 74% to GUI. Content editing (85.8%) and result checks (98.4%) prefer CLI; spatial layout prefers GUI at 58.4%. A matched SFT run with two separate tools scores 38.8%; the unified bash schema scores 46.0%.
Per-app APIs buy speed at engineering cost. The shell is already there, if the policy can decide when to use it. HybridCUA makes that decision a training target via b-star and the two CLI rewards, instead of leaving it buried in cloned traces. That is the gap versus CUA-Universe (VLM scoring, coarse feedback) and RecreationWorld (outcome-only rewards).
Bulk file edits, config writes, and verification fit CLI; visual layout and web pages stay on the GUI. The 7.2-point schema ablation also says tool packaging is not a prompt footnote.
This is a data-and-reward recipe, not a new backbone. The 9B model still trails EvoCUA-32B, and the Windows gain is 4.0 points. The pieces worth copying are the three trajectory types and the two CLI-aware rewards.
The authors flag the binding constraint: no usable CLI, a restricted shell, or OS-specific command semantics can make the routing policy fail. The two OOD suites do not cover app diversity, permission models, or long-horizon workflows. Data construction stays close to OSWorld's 11 apps; Calc alone is 26.8% of the RL pool.
Each reported number is a single run. One point on 361 tasks is three or four items; variance is unreported. Mean verifier scores include fractional graders, so they are not strict success rates. OSWorld-MCP reuses OSWorld's tasks and only changes the environment. The Windows +4.0 has no breakdown of correct PowerShell versus leftover GUI clicking. Teacher sampling used hand-written per-app CLI skills. The student never sees those skills, but the trajectory distribution does, so transfer to apps without such skills is untested.
Shell access also raises the blast radius of a single action. The ethics statement puts real-world safety and human oversight outside the study.