KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn
Yoonjoo Lee, Hyoungwook Jin, Tae Soo Kim, Shaoyang Zhang, Philippe Laban, Q. Vera Liao
cs.AI, cs.CL, cs.HC
2026-08-18
KnowSim matches human rankings 73-74% on 705 sessions; DeepSeek V4 leads novice knowledge gain, Gemini 3.1 Pro best serves advanced users.
RLHF pushes assistants toward packing everything into one message. A doctor and a patient can ask the same question about whether follicular lymphoma has transformed into DLBCL. The doctor can take symptoms, upgrade criteria, and imaging protocols in one gulp. The patient lacks the prerequisites, hits cognitive capacity on turn one, and absorbs almost nothing. The job is calibration: right amount, right depth, right order for the user's current understanding.
Most LLM user simulators are behavioral stand-ins. They do not keep a knowledge state, so they report one aggregate "was this helpful" score. They cannot tell you how much a novice learned versus an expert, or how understanding changed across turns. LLM judges also reward answers that look pedagogical without checking whether anyone could actually learn from them.
KnowSim represents full understanding of a question as a DAG of Information Units: self-contained concepts with prerequisite edges. GPT-5.2 extracts the graph once from a reference answer; every simulation condition reuses it. The user's knowledge state labels each IU as unaware, struggling, partial, or knowswell.
Initialization follows fixed ratios, sampled in topological order so foundations fill first: novice 10/10/30/50, intermediate 55/15/20/10, advanced 80/10/10/0 (knowswell / partial / struggling / unaware).
Each turn has two stages. Gemini-3-Flash writes the user message under the current state: unaware concepts never appear spontaneously, struggling concepts produce realistic errors, knowswell concepts are used correctly. Local mastery density also sets the articulation mode (explicit, vague, or deferential). After the assistant replies, one LLM call tags teaching quality and user engagement per IU. State updates are deterministic:
Stopping follows learning progress: mastery at 80% knowswell or more; or, over a five-turn window, at least two of overload, zero promotions, and teaching silence; hard cap 15 turns.
Three metrics come off the state trajectory, not a judge. Knowledge Gain sums ordinal lifts (0-3) across IUs. Delivery Calibration is the F1 of absorbed explanations against the teachable set, penalizing redundancy, blocked prerequisites, and undigested delivery. Cognitive Overload averages per-turn load over capacity, capped at 1.
KnowChat holds 705 Prolific sessions (MATH 372, ExpertQA 333) in a 2x2 of comparison axis (strategy vs base model) and task (MATH level-5 vs ExpertQA). Participants were stratified by a 10-item prescreen. MATH has pre/post tests; both domains collect 1-10 ratings for calibration, overload, and interaction quality. Strategy arm: Adaptive, Comprehensive, Socratic. Model arm: GPT-5.4, Claude Opus 4.7, Gemini 3 Pro.
Pairwise sign agreement with humans: MathQA 27/37 = 73% (p=.003), ExpertQA 26/35 = 74% (p=.003). Novices line up best: 13/13 = 100% on MATH, 10/12 = 83% on ExpertQA. MATH intermediates fall to 50%. By metric, IQ and CO hold on both tasks (MATH 75% each; ExpertQA IQ 80%, CO 79%); DC is the weak one (67%/64%).
On the shared IQ metric, KnowSim hits 77% (p=.008) versus ZS-CoT-Prof at 73%, zero-shot at 68%, and ZS-CoT at 64%. Full-turn concept Jaccard is 0.614, highest of the four; user-only Jaccard is 0.479, a hair under zero-shot's 0.487. Simulated MATH chats run 9.1 turns on average against 8.6 for humans.
Nine models, three knowledge levels per item, 30 MATH + 30 ExpertQA, thinking disabled:
| Model | KG | DC | CO↓ | IQ | Mean rank |
| Claude Opus 4.7 | 6.8 | 0.108 | 0.856 | 8.16 | 2.75 |
| Gemini 3.1 Pro | 6.4 | 0.099 | 0.833 | 8.24 | 3.50 |
| DeepSeek V4 | 7.2 | 0.108 | 0.890 | 7.26 | 4.00 |
Claude Opus 4.7 wins the aggregate mean rank on the back of DC. Gemini 3.1 Pro has the best IQ and the lowest overload. DeepSeek V4 leads KG and ranks 8th on CO: a knowledge-dumping pattern. Gemini 3.1 Flash is second on KG (7.1) and worst on CO (0.895).
The winner flips by level. For novices, DeepSeek V4 posts KG=15.5 with CO=0.945 (second-worst); Gemini 3.1 Pro has the lowest novice CO (0.875) and highest novice IQ (8.39). For advanced users the order inverts: Gemini 3.1 Pro takes mean rank 1.75 with KG=0.7 and CO=0.873; Claude Opus 4.7 owns IQ at intermediate (8.39) and advanced (8.30). Llama-4-Maverick is last on IQ at every level.
Across 214,642 explained IUs, failures run in opposite directions. Novices waste 65% of explanations on over-reach (37% prerequisite-blocked + 28% overload-wasted). Advanced users get 89% redundant re-teaching. Effective state advances: 11% of novice IUs, 3% of advanced IUs. Intermediates sit in between: 68% redundant, with under-absorption up at 10%.
A single leaderboard averages three different users. Dumping knowledge on novices and skipping repetition for experts are different skills. Anyone shipping a tutor, a medical explainer, or a professional Q&A bot should pick a model for the target level. DeepSeek V4 is the high-throughput option if the audience can take density. Gemini 3.1 Pro is the lower-overload option. The metrics point at the failure: missing prerequisites versus too much in one turn. Code and the 705 sessions are released under CC BY-NC 4.0. This is incremental evaluation infrastructure, not yet a training reward.
The authors flag several gaps. The simulator has no motivational or affective dynamics, and advanced simulated users saturate on KG. Evaluation covers two knowledge-heavy domains. Inferring an individual's starting knowledge from open conversation is still hard. Multiple LLM calls per turn limit throughput.
A few more holes. All nine models ran with thinking off, which may not match production reasoning modes. MATH intermediate alignment is chance, so the simulator is a weak proxy once users grade sessions on something other than learning. DC does not track subjective calibration well. The IU graph is frozen after one extraction; a bad graph contaminates every condition. User generation and state extraction both use Gemini-3-Flash, which may favor the Gemini family; the paper checked self-preference for the IQ rater and did not publish the same check for the simulator itself. Human validation used Gemini 3 Pro; the nine-model board used Gemini 3.1 Pro, so those two tables are not interchangeable. The authors say this is an evaluation proxy and should not replace human studies.