Work2Skill Turns Human GUI Demos into Reusable Agent Skills, Beating No-Skill Baselines
nex-agi · hf · 2026-10-12
Researchers from Shanghai Jiao Tong University, Fudan and others introduce Work2Skill, a framework that converts human desktop GUI experience into reusable skills for computer-use agents.
- WorkRecorder: infrastructure aligning screen recordings, mouse/keyboard events, and accessibility-tree data.
- WorkDemo-350: the first synchronized multimodal dataset of human desktop GUI demonstrations (350 screened demos) for skill induction and evaluation.
- Agentic skill induction: inspects evidence on demand to produce skills capturing workflow rules, step dependencies, and completion checks.
Across four execution models under shared settings, Work2Skill achieves the highest scores and success rates, improving mean scores by 4.67–10.54 points on MyPCBench Bounded Action and 3.11–7.50 points on a 286-task subset of OSWorld-Verified versus No Skill baselines. Code and data are open-sourced.
More from coding & agent
- Ice Cubes dev uses Codex to clear open-source repo backlog of bumps and fixes — Dimillian · 2026-10-12
- SEO team uses Grok bot groups as web crawler, sometimes beating ScreamingFrog locally — gaganghotra_ · 2026-10-12
- EXL3-QLoRA lets you fine-tune 2-8bpw quantized models, beating BNB NF4 on loss — Unstable_Llama · 2026-10-12
- Linus Torvalds is using Claude Code on a hefty Linux kernel performance improvement — PointmanW · 2026-10-12
- Claude-driven Blender 3D modeling from images makes this dev bullish on inspection gear — jwt0625 · 2026-10-12
- Theo vows T3 Code will never get native LaTeX support, sparking user debate — theo · 2026-10-12