Open-Source Screenshot Intent Classifier: Detecting When Users Want the LLM to See Their Screen
TheMoonMidas · x · 2026-09-02
@drstoney open-sourced a fine-tuning demo, screenshot-intent-classifier: a small classifier that reads user conversation history to decide whether a question is about something on their screen and whether to trigger the takescreenshot tool.
- Ships five variants: ModernBERT, Longformer, DistilRoBERTa, MiniLM, and DistilBERT Multilingual; trained on user-only history with left truncation for recent context, outputting a binary takescreenshot/noscreenshot decision.
- Usable as a tool-routing model by a text inference server, called on each user message.
Author's postmortem: users were confused about when the LLM peeked at the screen, so he abandoned the direction—simpler to let users take a screenshot and let the LLM analyze it. Repo includes dataset, training and QA scripts.
More from coding & agent
- Foldkit 0.155 Released: Unifies SSR Build Command and Adds Interaction Controls — samgoodwin89 · 2026-09-02
- Google SRE Author Submits PR to SREGym for Resource Stability Fixes — tianyin_xu · 2026-09-02
- Add Cloudflare.Telemetry native tracing layer to Alchemy — samgoodwin89 · 2026-09-02
- Qwen Reasoning Traces Show Strange Refusals and Self-Commands — wombweed · 2026-09-02
- Shopify open-sources Tangle; 0.8B model finetune beats GPT-5.6-sol — kieranklaassen · 2026-09-02
- LangChain adds agentic video understanding support, lowering token costs — hwchase17 · 2026-09-02