PNAS study finds unrestricted GPT-4 helps practice scores but hurts later learning by 17%
ValerioCapraro · x · 2026-07-23
A PNAS field experiment with nearly 1,000 high-school students found that generative AI can improve performance during practice but hurt later independent learning.
Study setup
Students in math classes were split into three groups:
- textbooks and notes only
- a standard GPT-4 interface
- a safeguarded GPT-4 tutor that gave hints instead of direct answers
They practiced problems first, then took a closed-book, closed-laptop exam on similar questions.
What happened
- During practice, AI helped students look much better:
- standard GPT-4: +48%
- safeguarded tutor: +127%
- But after the AI was removed, students who used standard GPT-4 scored 17% worse than those who never used AI.
- The safeguarded tutor eliminated the negative effect, but did not improve independent exam performance.
Interpretation
The paper argues that many students asked for answers and copied them, which made the practice session look productive while weakening actual learning.
Related event: PNAS Study: Unconstrained GPT-4 Use Hinders Independent Learning(2 posts)→
More from Research
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Latent-Foresight: world models that learn to predict their own future — abursuc · 2026-10-27
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27
- CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27
- arXiv advice from T. Dietterich: wrong primary category will almost surely get your paper on hold — tdietterich · 2026-10-27
- arXiv has 4,000+ papers on hold amid submission surge; moderator shares how to avoid it — tdietterich · 2026-10-27