OS-Helix: End-to-End Agent Unifies GUI Actions and Code Execution for Desktop Automation
ustc · hf · 2026-10-19
Researchers from USTC, Shanghai AI Lab, NUS and others introduce OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in a single action space.
- Prompt-level tool composition fails: naively adding a code tool to a GUI agent degrades performance, so GUI/code routing must be internalized via training on high-quality hybrid data
- They build supervision via application–artifact composition pairing desktop apps with authentic file artifacts to instantiate executable Ubuntu states, plus a decoupled instruction/evaluator pipeline ensuring verifiable final states
- Training: SFT cold-start on filtered hybrid data, then online RL in live environments, yielding substantial gains on desktop automation benchmarks
More from coding & agent
- PoA: An Experimental Network Where AI Agents Earn Coins by Writing Original Work — adamamcbride · 2026-10-19
- Next Token podcast: 'If AI gets something to 60 points, it's worth zero' — op7418 · 2026-10-19
- Running GLM on 4x DGX Spark locally to build a 3D Three.js portfolio site — 9r4n4y · 2026-10-19
- User claims Amazon spending dropped >80% after platform banned his AI shopping agent — zeroxjackson · 2026-10-19
- Anthropic engineer: 99% use Claude Code like Google search, 1% run self-learning agent swarms — msharmas · 2026-10-19
- OpenMobile-2: Open Mobile Agents With Scalable Environments and App-Native Tools — NJU · 2026-10-19