Simular's Ang Li shows the Autonomous Computer: vision agents plus reusable skills
AI Engineer · youtube · 2026-10-12
At the AI Engineer workshop, Simular founder Ang Li presented the infrastructure behind their computer use agent:
- Core approach: combine adaptable vision agents with reusable automation code, checking the environment at each step — the agent saved successful steps of finding a conference speaker as a "skill", replayed it next day, and fell back to model-driven exploration when a hiccup broke the workflow.
- Agent S trajectory: from accessibility-based control to visual grounding, multiple models, and parallel attempts.
- Demos: a coding agent opening a website, reading an accessibility snapshot and summarizing it; code replay and repair; legacy form automation and Android interactions.
- Infra: scaling from one agent on a personal desktop to fleets of cloud computers controlled via interface or API.
- Reliability framing: not one lucky success, but repeated task completion across varying prompts and environments; Q&A covered screen understanding, verification loops, irreversible actions, and OS restrictions.
The session remains an experimental direction: benchmarks and demos still leave much work for everyday computer use reliability.
More from coding & agent
- 817 structured cybersecurity skills turn Claude Code, Copilot and Cursor into senior security analysts — tom_doerr · 2026-10-12
- Harvard's CS2680 teaches modern AI agents from Claude Code down to KV cache and quantization — CurieuxExplorer · 2026-10-12
- The debugging sequence: how to investigate sudden AI quality drops step by step — goyalshaliniuk · 2026-10-12
- How to get all 3 free Anthropic Claude certifications in about 9 hours — Arindam_1729 · 2026-10-12
- One Prompt: Claude Code Publishes a Real Instagram Post via Custom MCP Server — Dull_Alps_8522 · 2026-10-12
- SoftwareGym synthesizes 182 app environments, lifting CUA OSWorld Pass@3 to 56.97% — Tian Jin · 2026-10-12