Effective and Scalable Math Support: Evidence on the Impact of an AI- Tutor on Math Achievement in Ghana
Owen Henkel, Hannah Horne-Robinson, Nessie Kozhakhmetova, Amanda Lee
cs.HC
2024-02-15
A school-level Ghana trial of the WhatsApp math tutor Rori finds a 3.01-point growth gap over 8 months (Cohen's d=0.36) among 477 students, at about $5 each.
Fewer than 15% of students in sub-Saharan Africa reach minimum math proficiency by the end of middle school. One-to-one tutoring works. It does not scale in low-income systems. Computer adaptive tutors have posted gains of 0.47 to 0.6 standard deviations, but fewer than 20% of West Africans have a home computer and reliable broadband. Phones do reach people: about 90% own a mobile phone and live under 3G coverage.
Randomized Teaching at the Right Level programs associated with Banerjee and Duflo reported about 0.28 SD, or roughly 1.5 extra years of school, at the cost of teacher training and operational change. Rising Academies, a school network in Ghana, packed similar pieces into Rori, a WhatsApp math tutor. The question is whether a chat tutor used during study hall can move scores at a cost a ministry could copy.
Rori follows current skill, not grade. The curriculum has more than 500 micro-lessons aligned to the Global Proficiency Framework. Each item has a short explanation and scaffolded practice. A wrong answer gets a hint, then a worked solution. Students type in natural language. NLP and an LLM parse attempts and open questions.
Eleven similar Rising Academies schools were randomized at the school level, six control and five treatment. Student-level randomization was not feasible. Treatment students used Rori twice a week for 30 minutes during study hall; teachers handled devices, not math. Control students kept regular math in the same period. Core class hours and curricula were identical.
The same 35-item test, covering grades 3-5 numeracy and algebra on that framework, was given in February 2023 and again in late August. 637 students in grades 3-8 sat the baseline; 477 sat both tests (241 control, 236 treatment). 160 left, mostly from uneven attendance.
Growth is endline minus baseline. Control mean 2.12 (SD 6.30), treatment 5.13 (SD 7.03), difference 3.01, p<0.001.
| Arm | Baseline | Endline | Growth |
| Control N=241 | 20.20 | 22.32 | 2.12 |
| Treatment N=237 | 20.29 | 25.42 | 5.13 |
Cohen's d, pooling SDs across arms and waves as Morris recommends, is 0.36. In education research that is moderate to large: Hattie marks 0.29 as moderate and in the range of a good teacher; Kraft treats interventions above 0.20 SD as large. The control gain of 2.12 is ordinary instruction plus retest. The extra treatment gain is described as about a year of schooling or more.
Baseline scores, gender, and age did not differ. Dropouts scored lower at baseline (18.53 vs 22.26). The reported d uses only completers, so that gap does not mechanically change it. Marginal cost is put at about $5 per student (LLM API, handset, 3G).
This is field evidence of an LLM tutor inside a low-income school day, not a lab demo. WhatsApp is the channel because computers are not. $5 next to 0.36 SD is a number a budget office can compare with shrinking class size or hiring tutors.
The design cannot split adaptive feedback from simply replacing unstructured study hall with structured practice. Treatment did not add an extra hour after school.
Eleven schools is a thin cluster sample. Unobserved school differences remain, including Hawthorne effects and household background. The network is used to routine testing, which may dampen observer effects. Cluster-robust standard errors are not reported.
One easy test for all grades creates a ceiling for older students, so growth may be understated, or Rori may mainly help easier items. Completers are stronger than dropouts. There is no dosage curve on time spent. The study covers year one inside one network. Table 1 lists 237 treated completers; the text says 236.