Mathematician's AI Review Experiment: 97.7% of 266 Auto-Generated Comments Flag Real Issues
littmath · x · 2026-08-27
Mathematician Daniel Litt published the full record of running Refine.ink on 19 of his papers: of 266 detailed comments, 239 were correct, 21 partially correct and only 6 wrong — 97.7% identified a real issue. A ChatGPT audit plus manual spot-checks turned these into 216 numbered erratum items. He calls the output "slop" but useful, noting AI often over-edits (adding a paragraph where one word would do).
By impact: 209 were local correctable errors (statement, proof step, formula, citation, hypothesis); the audit found 2 substantial theorem-preserving defects and 5 technical corrections to main results, with no fundamental failures. He adds that most errors came from badly propagated edits while optimizing, and that one known lemma error found by human peers (Jordan Ellenberg, Alex Smith) was missed by Refine. More errata await coauthor permission.
More from Research
- Interpretability paper incoming: in-depth look at testing explanations on modern LLMs — burny_tech · 2026-10-16
- Ex-OpenAI interpretability lead: neurons in trillion-param models are mostly legible now — CatAstro_Piyush · 2026-10-16
- Better-learning models tend to be more interpretable, argues Nick Cammarata — burny_tech · 2026-10-16
- MIT let Claude Code loose on a virtual island: it played for 30 hours with no task — CatAstro_Piyush · 2026-10-16
- Cornell and Amazon revisit the 16x pretraining compute gap: diffusion LMs need only 3-8x — LucaAmb · 2026-10-16
- AI trained on the fraction of human intelligence that fit the page — r0ck3t23 · 2026-10-16