AI2's TutorMoments benchmarks AI tutors on 520 real moments; human tutors score 0.458
allen_ai · x · 2026-10-13
Allen AI launched TutorMoments-Preview, a benchmark for whether AI tutors know when to scaffold versus when to push for rigor:
- 520 teacher-flagged key moments from real math tutoring sessions, split between scaffolding and rigor calls.
- Scores measure doing what the moment required without over-scaffolding; human tutors scored 0.458 (scaffolding), 0.182 (rigor), 0.496 (avoids over-scaffolding) — a naturalistic reference, not a ceiling, since the dataset concentrates on missed opportunities.
- Findings: Claude Sonnet 5.5 (low effort) was best at making students work through problems themselves when recommended; with explicit tutoring instructions, GPT-6 Sol (no reasoning) beat Claude Opus 4.8 overall.
- Includes an explorer comparing model vs human tutor responses alongside latency and cost.
Related event: Allen AI's TutorMoments Benchmark Finds LLM Tutors Overdo It for Students(5 posts)→
More from Models
- Google AI Overviews denied a person had died, renewing criticism of AIO for news — lilyraynyc · 2026-10-13
- User feeds Claude Opus GTA frame-by-frame screenshots since it can't watch videos — adonis_singh · 2026-10-13
- Gemini Deep Research on web is really slow today, user reports — gaganghotra_ · 2026-10-13
- Dual RAID10 NVFP4 setup loads GLM 5.3 Flash cold in ~1 minute — TheZachMueller · 2026-10-13
- Opus 5.5 stuns at SVG generation; Meta's Muse Wonder judged 3-6 months behind frontier — scaling01 · 2026-10-13
- Muse vs Opus 5.5: same 3D prompt costs $4.11 vs $99.43, big quality gap — iamaliveix · 2026-10-13