Self-Evolving Framework for GUI Assistants
HIT-TMG · hf · 2026-07-16
This paper introduces KnowAct-GUIClaw, a framework for personal GUI assistants aimed at solving two main issues:
- Lack of cross-platform GUI support: Difficult to migrate across Android, iOS, HarmonyOS, and Windows
- Absence of continuous self-evolution mechanisms: Inability to learn and improve from execution experiences
The authors propose the "Know Deeply, Act Perfectly" philosophy and designed the Know-Route-Act-Reflect framework:
- Know: The main agent uses accumulated interaction experiences and task knowledge for long-horizon task decomposition and allocation
- Act: Pluggable GUI sub-agents equipped with experiential attribution memory and a self-evolving skill library
- The system continuously saves user profiles and feedback to improve task decomposition and tool calling accuracy
Experiments across four system platforms show improvements in efficiency, accuracy, and cross-platform adaptability. Powered by the open-source Kimi-2.6, GUIClaw achieved 64.1% on the MobileWorld long-task benchmark, outperforming competitors including Seed-2.0-Pro and GPT-5.5. The framework also transfers well to different base models, boosting Kimi-2.6's performance by 8.5%.
More from coding & agent
- GitHub review bot hits its PR limit and forces a 39-minute cooldown — DanielLockyer · 2026-07-22
- Max reasoning effort appears to be mobile-only in Codex Remote, not desktop — GabGarrett · 2026-07-22
- A Reddit demo argues online stores should expose carts and pricing through MCP — gelembjuk · 2026-07-22
- Open-source AI SDK provider routes Vercel apps through a local Codex subscription — lgrammel · 2026-07-22
- Codex vs Claude Code: Which Is More Popular? — jxnlco · 2026-07-22
- CodeRabbit uses layers, diagrams and a chat agent to rethink code review — _jaydeepkarale · 2026-07-22