StudentBench: AI and human tutoring yield equivalent GRE learning gains
Curtis Northcutt, Inaara Hasmani, Kevin Feng, Trevor Khangi, Andreas Plesner, Jonas Mueller
cs.AI, cs.CY
2026-09-24
Across 2,383 GRE students, pooled AI tutoring matches expert humans (p=.015). Gemma 4 31B hits equivalence at $0.0052 per point, about 918× cheaper than a $75/hour tutor.
Most LLM leaderboards ask what the model can do unaided. Handshake AI asks the other question: given one hour with a student, can an off-the-shelf model produce learning gains matching an expert human tutor. The exam is the GRE. Quantitative is 27 questions in 47 minutes, Verbal 27 in 41, written by former ETS and Kaplan item writers so old public items are not in the mix. The post-test is a parallel form. Models never see it.
The authors frame this as the first condition for recursive human self-improvement: teaching technology has to raise human capability. Software scaffolding is kept thin on purpose, two low-guidance prompts (lesson planning, live chat), so the measurement tracks the model more than the harness.
2,383 students and 2,469 filtered sessions, drawn from about 70,000 random Handshake invites. After the pre-test, students are assigned to AI tutoring, live one-to-one video with a human tutor, or educational videos unrelated to the GRE. Inside the AI arm, 13 tutors (12 models; Opus 4.8 with reasoning off and extra-high counted separately) are balanced; students are not told which. Tutors see pre-test misses and never the post-test. Gains are ANCOVA-adjusted for pre-test score. Equivalence uses TOST at \(\pm 0.25\) pooled SD of gains (about \(\pm 4.09\) points combined); the entire 90% interval must sit inside the bounds.
A second study has 51 expert tutors run 2,028 pairwise reviews of lesson plans and practice items from 381 pre-tests. Conversational pedagogy is six fixed text rules over 1,971 AI and 135 human transcripts. Cost is recorded inference; humans are priced at $75 per hour. The platform, data, and code are public, with more than 175,000 student–AI messages.
Pooled across Quantitative and Verbal, AI tutoring matches human tutoring (\(p=.015\)). The adjusted AI−human difference is −0.58 points (90% CI \([-2.18, 1.03]\)), also inside the tighter \(\pm 0.20\) SD bounds (\(p=.023\)). Equivalence holds for Quantitative and fails for Verbal, where humans keep the highest whole-section mean. Versus the no-tutoring control, adjusted AI gains are +6.86 Quantitative, +5.47 Verbal, +6.15 combined, about 1.5–2 extra correct items out of 27.
| Condition | n | Adjusted gain [95% CI] |
| Opus 4.8 (x-high) | 158 | 16.0 [13.7, 18.4] |
| GPT-5.5 Pro (med) | 162 | 15.7 [13.4, 17.9] |
| Gemini 3.1 Pro (high) | 165 | 15.4 [13.1, 17.8] |
| Human | 140 | 14.9 [12.4, 17.3] |
| Gemma 4 31B (high) | 159 | 13.1 [10.7, 15.5] |
| Control | 190 | 7.8 [5.8, 9.7] |
In five of seven GRE domains the best AI mean exceeds the human mean. Six models pass individual equivalence tests. Gemma 4 31B costs $0.067 per session and $0.0052 per percentage point, against $4.81 for the human reference, about 918×. GPT-5.5 Pro costs $21.24 per session. Gemini 3.5 Flash matches GPT-5.5 Pro on gain at about 20× lower cost.
Experts prefer Anthropic on lesson planning and practice design. Students dispute GPT-5.4 mini answers on about 11% of graded Quantitative practice; models that experts like for practice design draw fewer disputes. Pedagogy-rule scores cluster by model family. On Quantitative, faster replies track more student messages, more messages track more first-try correct practice, and more correct practice tracks larger gains (all \(p<.002\)); the chain is not significant on Verbal. Median reply time runs from 1.9 s (GPT-5.4 mini) to 31.0 s (GPT-5.5 Pro).
Bloom's two-sigma problem asked how to make one-to-one tutoring cheap. This is a checkable comparison: on a high-stakes GRE post-test, general-purpose LLMs with thin scaffolding already match expert humans on average gain, and the cheapest open-weight tutor puts the cost per point under a cent. Leaderboards that only score the model unaided miss this axis.
Equivalence is the pooled mean, not every model, section, or student. Verbal still favors humans. The planning leaderboard and the gain leaderboard do not match: Opus plans well, Gemma is cheap enough and equivalent. Latency may be an under-counted teaching variable.
Gains are immediate; months-later retention is unmeasured. There is no solo GRE-practice control, so dialogue cannot be separated from simply doing problems. Humans taught on video, AI on typed chat, so pedagogy-rule scores are not cross-modal. The sample is English-reading adults on Handshake in a paid study, not a classroom or another language. Figure 3 ranks expert preference and rule hits, not learning. Per-tutor equivalence tests are uncorrected for multiple comparisons. The RHSI framing has three conditions; this paper tests only the first.