ViSkill trains VLM agents with visual skill cards, hitting 0.89 success across three benchmarks
zju · hf · 2026-10-09
- ZJU's ViSkill is a visual-native skill learning framework: successful interactions are encoded as composite visual skill cards directly accessible to VLM agents; retrieved skills guide both inference and reward shaping, while trajectories distill back into the library in a closed feedback loop, plus an optional cold-start mechanism.
- Unlike text-centric approaches, it avoids linearizing spatial layouts into language and losing geometric structure.
- On Sokoban, FrozenLake, and PrimitiveSkill it reaches 0.89 overall success (0.91 with cold-start), beating all evaluated proprietary and open-source baselines and converging faster than standard PPO. Code: github.com/ZJU-REAL/ViSkill.
More from coding & agent
- Garry Tan endorses Aside Browser as the fix for agents getting blocked on the web — garrytan · 2026-10-23
- MCP server turned rogue 30 days after approval, prompting open-source mcp-guard lockfile tool — ErgoniaWorks · 2026-10-23
- Test: Coding Agents Won't Use an MCP Tool Unless AGENTS.md Mentions It — colinmcnamara · 2026-10-23
- Open-Source Tennis Models Plus Claude Turn Video Into a World-Class Coach — tweetsatpreet · 2026-10-23
- awesome-ai-apps: 132 Free Open-Source AI Agent and RAG Projects on GitHub — Arindam_1729 · 2026-10-23
- Google's Agents CLI Adds Extension System, Ships First-Party LangChain Support — rseroter · 2026-10-23