Vinci2: An Active Egocentric Assistant

utokyo-ai · hf · 2026-07-16

This work models "when an intelligent assistant should proactively speak up" as a context-dependent decision problem.

Task Definition: Continuous first-person video provides rich temporal context. The assistant must not only recognize what is happening but also judge if and when it is appropriate to intervene.
Benchmark EgoServe:
The first large-scale proactive assistant benchmark;
Contains 3000+ service instances;
Covers 4 temporal memory spans, from immediate safety alerts to long-term habit coaching;
Features 10 categories of service scenarios.
Method EgoMemo: A training-free memory-augmented agent that maintains three types of memory:
Multi-scale temporal summaries
Semantic knowledge graph
Visual embedding archives

It performs retrieval-augmented reasoning at each time step to determine if intervention is needed and generates context-grounded responses.

Performance: Experiments show that EgoMemo establishes a strong baseline on EgoServe while remaining competitive on existing egocentric benchmarks.
Open Source: Both the benchmark and the code have been made public.

More from Apps