Looking for an AI that can learn from screen video to operate a system over time
angrykirby · reddit · 2026-10-11
A Reddit user has built the control loop for a screen-operating agent—reliable programmatic inputs plus screen capture as visual feedback—but needs a model that can learn from visuals, remember what it has seen and been taught, and improve at operating the system over time.
Models tried so far fall short on memory and often act without consulting the visual input first. He's asking the community for recommendations on stronger vision-memory-action setups.
More from coding & agent
- Ethan Mollick's new book charts 30 months of AI progress, from cyborg to conjurer work — rseroter · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang details Vera Rubin agentic stack: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- Capy 0.5.0 lets Claude Pro/Max subscribers reuse their plan quota, adds OIDC cloud access — garrytan · 2026-10-23