HABIT-Bench: Best agent memory system scores just 33.61% at inferring user habits
SJTU · hf · 2026-10-28
SJTU's LUMIA Lab and collaborators introduce HABIT-Bench, evaluating whether LLM agent memory systems can infer implicit user habits from sparse dialogue evidence and apply them across long-term interactions.
- Synthetic user histories built from task-oriented dialogue data and controlled habit graphs, spanning food, finance, software, and travel across hundreds of sessions
- Four-choice tasks include temporary exceptions, habit changes, and unaccepted suggestions
- Eight memory methods evaluated with Qwen3-235B-A22B and GLM-5.3-Flash: the best reaches only 33.61% macro accuracy (25% random), and performance stays limited even with annotated evidence supplied directly
Key takeaway: access to past interactions does not equal the ability to infer and appropriately use user habits. Code and dataset are open-sourced.
More from coding & agent
- Harrison Chase on self-improving agents: LangChain aims to be the model-agnostic all-in-one platform — hwchase17 · 2026-10-28
- Qiaomu UI plugin gives coding agents 13,561 design references to clone quality landing pages — vista8 · 2026-10-28
- Recurse: open-source minimal research agent harness built on Pi Durable — chaitjo · 2026-10-28
- Going multi-LLM: Sonnet 3.7 for hard tasks, DeepSeek for the rest, which gateway stack? — Extension_Lake4065 · 2026-10-28
- Anthropic prompt tip: give Claude image tools to inspect dense charts and diagrams — minchoi · 2026-10-28
- Give multi-agent teams a time budget: parallel work cut 30 minutes to ~15 — minchoi · 2026-10-28