MINTEval, accepted at NeurIPS, shows LLM agents fail at tracking evolving contexts
EliasEskin · x · 2026-10-02
MINTEval, a benchmark accepted to NeurIPS, tests LLM agents and memory systems in long-horizon, continually changing environments (Wikipedia pages, Git repos). Key findings:
- Avg. 86 context updates with interference; 5 question types incl. long-range lookback and multi-target reasoning
- 4 realistic domains: state tracking, multi-turn dialogue, Wikipedia revisions, GitHub commits
- Avg. 138.8k tokens per instance (up to 1.8M); 95.6% human-verified QAs
Across 7 models, agents struggle to track changes, recall precise details, and combine conflicting updates: models confuse old vs. new information, memory systems insert redundant entries, and retrieval often fails—rooted in limitations of retrieval and memory construction.
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Effective HTML: 4.2k-star agent skills turn wireframes into self-contained interactive prototypes — tom_doerr · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16