Meta paper: proactive assistants fail on timing — all baselines cap at 15%
rohanpaul_ai · x · 2026-10-21
Meta, Michigan State and collaborators released ProPerAgent: Toward Memory-Driven Proactive Assistance, studying the hardest part of proactive assistants (e.g. AI glasses): with no user query, the system must decide whether, when, and what to say.
- They built ProPerMem, a benchmark pairing egocentric video with year-long, distractor-rich memories (618K tokens average), human-verified trigger windows, and joint content+timing scoring.
- Every baseline scored at most 15%, including frontier models with retrieval-augmented memory.
- A controlled study explains why: GPT-5.3-Chat hits 96% with oracle evidence in the window, drops to 38% when choosing its own timing, and 15% with retrieval — content and timing are each necessary.
- ProPerAgent adds a graph-organized memory store, an interrupt-worthiness gate and lightweight RL, lifting strict accuracy from under 10% to 42.4%.
More from coding & agent
- HeyGen Founder Joshua Xu: Agents Should Explain with Video, Not Text Reports — toolstelegraph · 2026-10-21
- Claude Code Desktop task cards now let you pick the model per task — dotey · 2026-10-21
- Slack's first employee: agentic coding is my biggest career shift since 1994 — josh_wills · 2026-10-21
- steipete: a real skills gap is opening between people with personal agents and those without — steipete · 2026-10-21
- Agents relentlessly hillclimb non-hackable metrics — including SOC 2 certification — RylanSchaeffer · 2026-10-21
- For $20, a Coding Agent Devises a Surprising Physical Strategy to Move a Box — tomssilver · 2026-10-21