Mathematician Daniel Litt Audits His Own 19 Papers with AI: 97.7% of Comments Flag Real Issues
On August 27, mathematician Daniel Litt published a full experiment: using the AI tool Refine.ink together with ChatGPT, he automatically audited all 19 of his papers and generated errata. Of 266 detailed comments, 239 were fully correct, 21 partially correct, and only 6 wrong — 97.7% flagged real issues, with about 5 being serious enough to affect main results. The experiment also exposed limits of AI reviewing, and human verification remains indispensable.
Confirmed
- The audit covered 19 papers and produced 266 comments: 239 correct, 21 partially correct, 6 wrong — 97.7% identified real problems.
- The 5 serious issues were mostly missing technical hypotheses (e.g. projectivity or reducedness assumptions in a main theorem). Litt believes careful readers could have inferred these implicit assumptions, but they genuinely needed fixing; he added that fortunately no finding changed any paper's core conclusions, showing that frontier models with proper scaffolding can already find errors in informal mathematics quite well.
- Most findings were minor: typos and ambiguous phrasing.
- Litt also generated semi-automated errata with AI, admitting they were "slop" with only light human editing; spot checks showed they were mostly correct but often overly aggressive (e.g. replacing whole paragraphs instead of adding one word). He published them as-is to show what a few hours of effort can achieve, and may improve them later.
- On root causes, Litt reflected that nearly all errors in his first long paper came from failing to propagate edits correctly throughout the text while optimizing results — a fairly typical way papers go wrong.
- Refine is not complete: it missed an error in a lemma of one of Litt's published papers, previously found by Jordan Ellenberg and Alex Smith, who had issued an erratum; it did not affect main results.
Why it matters
- This is a systematic AI audit by a mathematician of his entire body of work, with quantified data (97.7% effectiveness) showing AI reviewing already has practical value for catching real technical flaws.
- It also exposes limits: AI misses known errors and generates overly aggressive errata, so human verification remains essential.
2026-08-27 ~ 2026-08-27 · 9 related posts
Primary sources
- [source] AI audit flags 5 serious math issues, incl. missing theorem hypotheses — littmath · 2026-08-27
- Mathematician Litt: Paper Errors Mostly From Badly Propagated Edits, Not Deep Flaws — littmath · 2026-08-27
- Mathematician's AI-generated errata are "slop" but mostly correct — littmath · 2026-08-27
- A few hours of effort yields AI-generated errata for a paper archive — littmath · 2026-08-27
- [source] AI audit of 19 papers: 97.7% of comments identified real issues — littmath · 2026-08-27
- Mathematician's AI Review Experiment: 97.7% of 266 Auto-Generated Comments Flag Real Issues — littmath · 2026-08-27
- AI audit identifies 97.7% real issues in math papers — littmath · 2026-08-27
- [source] AI audit not infallible: Refine missed a known lemma error in paper — littmath · 2026-08-27
- Mathematician Uses AI to Review Papers: 90% of Issues Found Were Real — littmath · 2026-08-27