StudentSim: Training LLM-based Student Simulators
Ke Yang, Chenglong Wang, Michel Galley, Chandan Singh, Jeevana Priya Inala, ChengXiang Zhai, Jianfeng Gao
cs.CL
2026-09-02
StudentSim's pooled-then-specialized 4B LoRA simulators hit chess F=0.51 and R=0.91, beating GPT-5.4; a tutor trained against it scores 90.5% accuracy.
An adaptive tutor needs two signals: how this student answers on their own, and whether a different explanation moves them. Real-student feedback is slow, expensive, and thin. In the L2 writing corpus the median learner has three essays; more than two thirds have five or fewer.
Existing simulators cover one side. Knowledge-tracing models and Maia2-style behavior predictors fit how people move or choose, and they have no channel for a tutor's natural-language explanation. A prompted LLM will follow that explanation fluently and still miss the competence and error signature of the person it is supposed to be. Tutor training needs both properties at once: look like this student, and be teachable.
StudentSim scores those properties separately. Behavioral fidelity F asks how well the simulator's answer on a held-out problem matches that student's own record: top-1 move accuracy in chess, error-rate and error-type profile in L2 writing, and whether a four-way multiple choice matches the option the student picked in math. Guidance responsiveness R asks, given the student's wrong answer and a tutor turn, whether the simulator updates to the canonical correction that turn is steering toward: the engine move, the teacher-corrected fragment, or the right math answer. F is the starting state. R is whether the simulator can be taught. A high-F, low-R model is a static clone. A high-R, low-F model follows advice from the wrong person.
Records are split per student into single-turn pairs (problem, answer) and multi-turn tuples (problem, wrong answer, guidance, canonical fix). Chess comes from Lichess games in May 2025, L2 from EFCAMDAT essays with teacher marks, math from a foundational-assistance corpus of student answers. StudentSimEval freezes 60 students (30 chess, 15 L2, 15 math) and scores every method on the same held-out slices.
Training is two-stage LoRA on Qwen3-4B-Instruct. Stage 1 pools every student in a domain so the base learns shared mistakes and how answers move after guidance, with a 0.20 multi-turn mix. Stage 2 continues on one student's records, one adapter each. Chess uses 100 players and 100,000 records in Stage 1, then 30 players with 1,000 records each; L2 Stage 2 has only 73 records per learner. Decoding is greedy at temperature 0. Headline numbers average three independent training seeds.
Guidance sources are asymmetric. L2 canonical fixes are real teacher spans; the tutor turn is rendered from point-based or rule-based templates. Chess and math have no teacher commentary, so an LLM writes the tutor turn under a fixed style template, while the target correction is pinned in advance by the engine or an audited answer key. The LLM chooses wording, not the destination.
A chess tutor RL study freezes StudentSim as the reward. The tutor is Qwen3-VL-8B reading a board image plus text, optimized with GRPO. Reward is Stockfish quality of the simulator's revised move, gated by two linear probes on the tutor text: whether the teaching style matches the intended mode, and whether the explanation names a wrong square or a hallucinated piece. That reward uses the Stage-1 pooled simulator, not a per-student adapter. Controls are the SFT tutor with no RL, and GPT-5.4 prompted as the student.
| Domain | Metric | Naive | GPT-4o | GPT-5.4 | StudentSim |
| Chess F (30) | top-1 move | 0.4535 (Maia2) | 0.2163 | 0.2316 | 0.5150 |
| L2 F (15) | error profile | 0.5130 (Qwen3-4B) | 0.4718 | 0.5141 | 0.5624 |
| Math F (15) | 4-way choice | 0.4919 (Qwen3-4B) | 0.5121 | 0.6121 | 0.6384 |
| Chess R | corrected move | 0.2721 (Maia2) | 0.7655 | 0.7186 | 0.9067 |
| L2 R | fragment rewrite | 0.0200 (Qwen3-4B) | 0.3883 | 0.5950 | 0.6417 |
| Math R | corrected answer | 0.6132 (Qwen3-4B) | 0.6940 | 0.7099 | 0.9181 |
Chess fidelity is about 6 points above Maia2 and 28 above GPT-5.4. Teachability is the real split: 0.91 against Maia2's 0.27 and GPT-5.4's 0.72. Maia2 has no language channel, so R sits near a no-guidance floor. GPT-5.4 follows advice and cannot impersonate a specific player. GPT-4o's chess R (0.77) is even higher than GPT-5.4. Math fidelity is only about 2.6 points above GPT-5.4; R jumps from 0.71 to 0.92. L2 gains are narrower on both axes.
On one held-out board, three players play e4, e3, and Bg5. Maia2 maps all three to the rating-modal e4. GPT-5.4 misses all three. StudentSim matches all three. In a Socratic chess case the prompt asks for "a forcing queen move that gives check from the queenside" and never names a square. StudentSim plays f8b4; GPT-5.4 plays the wrong queen move f8f4; Maia2 repeats the original blunder.
Replace Stage-1 pooling with one player's 1,000 records repeated 100 times and chess F falls from 0.513 to 0.460, R from 0.900 to 0.828. The population prior supplies structure that one sparse student cannot.
Eight competitive players, 74 blind annotations:
| Condition | Accuracy | Guidance (1-5) | Personalization (1-5) |
| No RL | 75.7% | 2.99 | 2.80 |
| GPT-5.4 student reward | 71.6% | 3.08 | 2.42 |
| StudentSim reward | 90.5% | 3.31 | 3.93 |
GPT-5.4 as the student reward drops accuracy to 71.6%, below the no-RL 75.7%, with more severe factual errors. StudentSim lifts accuracy to 90.5% and personalization from 2.80 to 3.93. The triply annotated subset and the 2000+ Elo raters keep the same order.
Prompting a frontier model to play a student looks usable on teachability, fails on fidelity, and can make the tutor worse when used as an RL reward. A 4B two-stage LoRA lands in the high-F, high-R corner in all three domains. Code and splits are public, so later methods can score on the same held-out records.
This is not a best-tutor claim. The chess RL loop is a proof of concept: is simulator feedback informative enough to improve a tutor. Fidelity gains on L2 and math are incremental. The portable pieces are the evaluation protocol and the decision to require F and R together.
The conclusion is explicit about the next gap: the metrics capture a state plus a one-step update, not how a student acquires, retains, or forgets knowledge across many turns.
Chess and math tutor text is LLM-written under templates. R measures whether the simulator reaches a pre-pinned canonical answer, not whether a classroom student would move that way. L2 Stage 2 has 73 records per learner and a median of three essays; the fidelity edge is thinnest there, in line with the data. Tutor RL is chess-only, because Stockfish supplies a reward independent of the simulator; L2 and open-ended math still need free-form rewards. Real student learning outcomes are unmeasured. L2 and math have no Maia2-like specialist, so the reference is mostly GPT. The human study is 8 raters and 74 annotations. A large share of the personalization score comes from the style probe, and the RL reward uses the pooled Stage-1 simulator, not the per-student Stage-2 adapters.