Microsoft finds LLMs lose track of user intent when conversations evolve
microsoft · hf · 2026-07-24
Static benchmark wins do not carry over when user intent changes
Microsoft Research studies collaborative LLM agents in dynamic conversations, where user intent is revealed, revised, and redirected over multiple turns rather than stated upfront.
The key idea
- They convert static, single-turn tasks into multi-turn evolving-intent conversations.
- The framework preserves the original evaluation protocol, so existing benchmarks can be reused without new annotations.
Main finding
- Across multiple tasks, models that perform strongly in static settings suffer substantial drops when intent evolves.
- The result suggests a fundamental gap: today’s LLMs still do not reliably track and act on changing user intent, even though that skill is critical for future collaborative agents.
More from Research
- Open-source decision models sold their own base in Red Alert 2 — then learned until no human could beat them — heathercmiller · 2026-10-27
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- World Model Assigns Weight and Friction to Objects, Then Simulates with a Physics Engine — jonstephens85 · 2026-10-27
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- Mathathon 2.0 launches with Terence Tao's SAIR, $1.5M in compute for teams — _sathvikr · 2026-10-27
- The 'sweet lesson' is real: two kinds of ML problems need opposite strategies — mike64_t · 2026-10-27