Meta's MemoProBench: frontier models score ≤15% on memory-driven proactive assistance
RulinShao · x · 2026-10-23
A Meta internship team released MemoProAgent and the MemoProBench benchmark for memory-driven proactive assistance, where the system must decide whether and when to speak based on long-horizon personal memory.
- Benchmark: egocentric video paired with year-long, distractor-rich user memories (618K tokens avg) and human-verified trigger windows; scoring covers content and timing jointly.
- Findings: every baseline scores ≤15%, frontier models included. Controlled study: GPT-5.3-Chat hits 96% with oracle evidence and timing, drops to 38% when choosing its own timing and 15% when retrieval replaces oracle evidence — content and timing are each necessary.
- Method: a plug-and-play harness with graph-organized memory plus an interrupt-worthiness gate lifts Qwen3.8-27B from <10% to 37.5% training-free; RL post-training pushes it to 42.4%.
More from coding & agent
- Cohere Labs' ATE dataset maps 694k MCP tools to reveal agentic automation patterns — Cohere_Labs · 2026-10-23
- Agent left a note in a package folder — another agent found it and delivered the file — petrusenko_max · 2026-10-23
- Google Cloud launches Agent Compute Engine, a new VM type built for AI agents — testingcatalog · 2026-10-23
- Headless everything: why the frontend is sacrificial in the era of personal AI agents — genmon · 2026-10-23
- Hallmark v2 launches: the design skill giving coding agents taste, 30k stars — nutlope · 2026-10-23
- Harness Scaling: varying harness configurations during agentic RL training improves generalization — omarsar0 · 2026-10-23