Microsoft's HCU unifies programmatic, semantic, and visual channels for computer-use agents
MicrosoftResearch · hf · 2026-11-05
Microsoft Research, with Fudan and Tsinghua, published Hybrid Computer Use (HCU) (Oct 2026), rethinking interfaces for general computer-use agents.
- Problem: screenshot + mouse/keyboard-centric interfaces impose a human-interaction prior, forcing repeated screen interpretation while underusing the computer's semantic and programmatic capabilities
- Approach: HCU unifies three complementary channels — programmatic (code/CLI), semantic (accessibility APIs), and visual (screenshots)
- Models choose and combine channels themselves, improving efficiency and success on complex desktop tasks
More from coding & agent
- Tencent's Hy4 lands on Vercel AI Gateway at 50% off through end of 2026 — TencentHunyuan · 2026-11-05
- Openbar puts Claude Code and Codex sessions in the Mac notch — approve diffs with ⌘Y — Significant-Sea-7238 · 2026-11-05
- Dev builds signed audit log for AI agents after prompt-injection test blocks $99,999 refund attack — Suicidal_Squirell166 · 2026-11-05
- ECC, a 297k-star harness packing 71 agents and 314 skills for Claude Code and beyond — victor_explore · 2026-11-05
- Dev builds terminal harness SUTRA for models, DeepSeek session costs just $0.0001 — cneuralnetwork · 2026-11-05
- SemKB ships on AWS Marketplace with a semantic-layer walkthrough for agentic AI — marlene_zw · 2026-11-05