Shelley now takes screen recordings as prompts — agent fixed a bug in one shot
davidcrawshaw · x · 2026-10-01
davidcrawshaw shares and comments on an exe.dev blog post: Shelley can now take a screen recording as a prompt — hit record, talk through your problem, click around, and the agent gets the transcript plus matching video as its prompt. He argues features like this can now be built into any software and should be OS-level and cross-app.
Blog highlights (author Philip Zeyliger):
- Shelley's new trick: instead of typing a prompt, record your speech and window/screen; on stop, Shelley combines the transcript with the matching video and hands it to the agent as the prompt.
- Example: Shelley had a bug where sending "Compact and Send" and navigating to a different conversation would revert to the original one. Rather than describe it, the author started a new conversation and recorded a repro video (unedited except shortcuts/URL bar cut off, with commentary and sped up).
- The recording is painful to watch — all the ums and figuring out what to click — yet the author's next message was just "push," and the agent reproduced from what it saw and fixed the bug in one shot, typing nothing.
Conclusion: give screen recording a shot.
More from coding & agent
- One prompt, 30 minutes: NVIDIA VSS Blueprint 3.3 builds production-line vision AI agents — NVIDIAAI · 2026-10-01
- Google AI proposes RRSI to stop recursive self-improving agents from overfitting benchmarks — burkov · 2026-10-01
- DevDay demo: Codex builds Minecraft for 30-year-old Game Boy hardware via ModRetro plugin — pvncher · 2026-10-01
- Full Talking Video From One Image in 30 Minutes for ~$5 — gorkem · 2026-10-01
- tldraw launches ChatGPT plugin: sketch your app's logic and Codex builds it — DavidKPiano · 2026-10-01
- Storing Claude's Memory Inside a Notes Repo With Auto-Commits — theshawwn · 2026-10-01