OSWorld-Persona: a 3,259-label benchmark for evaluating when agents should proactively help
EchoShao8899 · x · 2026-11-16
EchoShao8899 introduces OSWorld-Persona, a benchmark for evaluating whether an agent knows when to proactively offer help, built by turning computer-use recordings into personalized support examples.
- Built on OSWorld 2.0: 108 sessions, 6,558 key frames, 3,259 labeled proactive-support decisions, and 4 personas with different assistance preferences
- The same screen can call for different decisions depending on the user
- On-device privacy: local Observer, skipping redundant frames, observation over adaptation; KV-cache/prefix-cache co-design across main and speculative draft models with fast train/inference switching
- Five-step pipeline: recording collection, change-detection key frame selection, input construction (persona + rolling observation), two-stage labeling (perceive screen, decide whether to help with rationale), chronological per-persona train/test split
More from coding & agent
- Google Chrome hiring exec to build the agentic browser era — laparisa · 2026-11-16
- Cursor 'kinda dead' 3 months post-acquisition, says HF's Niels Rogge — NielsRogge · 2026-11-16
- DJI Mic Mini recorder app built with Codex in 3-4 prompts, shared by OpenAI Devs — OpenAIDevs · 2026-11-16
- Polestar Formula 2030 concept UI rebuilt as installable shadcn components — shadcn · 2026-11-16
- User's $300 subscription generated $7,492 of usage — 25x value on Codex and Claude — theteknosaur · 2026-11-16
- Your agent's inference endpoint sees all its memory: treat it as a data boundary — krishnan · 2026-11-16