OSWorld-Persona benchmark: deep personalization boosts proactive-help timing macro-F1 by 2.4-3.0x
EchoShao8899 · x · 2026-11-16
Researchers built OSWorld-Persona, a benchmark for evaluating whether an agent knows when to proactively help:
- Built on OSWorld 2.0 via a pipeline turning computer-use recordings into personalized support examples: 108 sessions, 6,558 key frames, 3,259 labeled proactive-support decisions, and 4 personas with different assistance preferences.
- The same screen can call for different decisions depending on the user.
Deep Personalization improves timing macro-F1 by 2.4–3.0x: Qwen-3.5-9B 0.186→0.483, Qwen-3.5-35B-A3B 0.183→0.433, Nemotron-3-Nano-Omni 0.118→0.349.
Contextual memory built by an on-device Observer further improves decisions about when to help.
Related event: OSWorld-Persona Benchmark Evaluates When AI Agents Should Proactively Help(2 posts)→
More from coding & agent
- W&B Sweeps adds Optuna and Ax: pick your HPO engine with one line of YAML — coreweavedevs · 2026-11-17
- Vibe Check: Gemini 4 Carbon Codes Better Than Argon but Trails Fable 5.2 on Judgment — every · 2026-11-17
- Gemini Managed Agents Adds Self-Hosted Sandboxes: Google Runs the Loop, You Run the Tools — _philschmid · 2026-11-17
- How to Pick Between Gemini 4 Carbon and Hydrogen: Plan With Carbon, Execute With Hydrogen — _philschmid · 2026-11-17
- An interactive O'Neill cylinder 3D world in one 1064 KB HTML file, no assets or libraries — NathanWilbanks_ · 2026-11-17
- An O'Neill cylinder 3D world in a single 1064 KB HTML file — no images, models, or libraries — NathanWilbanks_ · 2026-11-17