EMNLP 2026 papers show LLM rankings shift with user knowledge level in multi-turn evals
IanArawjo · x · 2026-10-25
Researcher yoonjoole2 will present two papers on LLM user simulation at EMNLP 2026:
- KnowSim: a user simulator that tracks and updates what a user knows as the conversation unfolds, enabling evaluation of information calibration over multi-turn interactions. The core claim: assistants can be correct and comprehensive yet poorly calibrated to the user—too hard for novices, redundant for experts. Across 9 LLMs, model rankings changed with the user's knowledge level.
- User Role Vector: investigates assistant bias in LLM user simulators via internal representations and activation steering.
The author is also on the job market, targeting user simulation, interactive AI evaluation, and oversight for agentic systems.
More from Research
- Hinton's Bank-Teller Insight: How Dropout Helped AlexNet Win ImageNet 2012 — SumitGup · 2026-10-26
- KnowSim at EMNLP 2026 simulates how users with varied knowledge levels seek information — tianshi_li · 2026-10-26
- dair-ai weekly picks: object-oriented agent harnesses, cache engineering lead — dair_ai · 2026-10-26
- CMU paper shows retrieval skew fools RAG into picking frequent near-miss answers; JCA fixes it — pratyushmaini · 2026-10-26
- OpenAI built an 'offline world' of cached web content to train frontier RL runs like GPT-7 — daniel_mac8 · 2026-10-26
- EMNLP paper: retrieval benchmarks like FreshStack stay reliable under temporal drift — mrdrozdov · 2026-10-26