Allen AI's TutorMoments Benchmark Finds LLM Tutors Over-Help Students
allen_ai · x · 2026-10-13
Allen AI (Ai2) released TutorMoments-Preview, an eval testing whether LLM tutors help students solve problems themselves or do too much for them.
Method
- 520 teacher-identified key moments from real math tutoring sessions, split evenly between moments calling for scaffolding and moments calling for a push for rigor
- Score = share of moments where the model did what the moment called for without over-scaffolding
- Two prompt conditions: a plain prompt ("tutor well") vs. an evaluation-aware prompt spelling out the scaffolding-vs-rigor trade-off
Key findings
- Frontier models including Astra and Opus 5.5 generally scored much better with the detailed, evaluation-aware prompt than with "tutor well" alone
- Human tutors at the same decision points: 0.458 appropriate scaffolding, 0.182 appropriate rigor, 0.496 avoiding over-scaffolding (dataset deliberately flags missed opportunities, so humans are a naturalistic reference, not a ceiling)
- Core insight: LLMs are trained to be "helpful assistants" and default to giving answers instead of diagnosing support needs through questions
Related event: Allen AI's TutorMoments Benchmark Finds LLM Tutors Overdo It for Students(5 posts)→
More from Models
- Muse Wonder defended as a real code model, backed by 10,565 commits in a year — ren_hongyu · 2026-10-13
- Early tester: Meta's new "Watermelon" model aces trick questions and self-checks code — kimmonismus · 2026-10-13
- Researcher predicts truth and morality will become mathematical artifacts on the path to superintelligence — ryunuck · 2026-10-13
- Decision Index is facing MTEB's benchmark-gaming problem, its author says — antoine_chaffin · 2026-10-13
- Half of Google's AI releases stuck in Experimental/Preview state, community mocks quality — scaling01 · 2026-10-13
- Anthropic accused of pressuring cloud providers to pull retired Sonnet 3 models for no clear reason — repligate · 2026-10-13