Microsoft finds LLMs lose track of user intent when conversations evolve
microsoft · hf · 2026-07-24
Static benchmark wins do not carry over when user intent changes
Microsoft Research studies collaborative LLM agents in dynamic conversations, where user intent is revealed, revised, and redirected over multiple turns rather than stated upfront.
The key idea
- They convert static, single-turn tasks into multi-turn evolving-intent conversations.
- The framework preserves the original evaluation protocol, so existing benchmarks can be reused without new annotations.
Main finding
- Across multiple tasks, models that perform strongly in static settings suffer substantial drops when intent evolves.
- The result suggests a fundamental gap: today’s LLMs still do not reliably track and act on changing user intent, even though that skill is critical for future collaborative agents.
More from Research
- Lab monkeys at $30-80K each: a 4-year breeding cycle is the real preclinical drug bottleneck — zebird0 · 2026-10-27
- OpenAI Foundation funds UNC tumor antigen project, hiring senior experimental scientist — iskander · 2026-10-27
- Steering attention heads toward honesty: up moral patienthood, down psychopathy — edelwax · 2026-10-27
- Are transcoders overkill? New paper argues plain neurons suffice for circuit analysis — burny_tech · 2026-10-27
- EMNLP paper GENIE: a fine-grained metric locating where LLM stories are novel — gregd_nlp · 2026-10-27
- Hashed n-gram value embeddings push GPT to 3.2115 loss, beating 30M track record — RobertTLange · 2026-10-27