AI audit not infallible: Refine missed a known lemma error in paper
littmath · x · 2026-08-27
Evaluating the completeness of the Refine reports, Daniel Litt noted they are definitely not perfect. The tool missed an error in a lemma in one of his published papers, previously found by Jordan Ellenberg and Alex Smith (and subject to an erratum, though it didn't affect a main result). This indicates that while AI audits are highly effective, human review is still necessary to catch missed nuances.
More from Research
- Interpretability paper incoming: in-depth look at testing explanations on modern LLMs — burny_tech · 2026-10-16
- Better-learning models tend to be more interpretable, argues Nick Cammarata — burny_tech · 2026-10-16
- 'Interp was mostly hopeless before — now it's automatable': Cammarata on interpretability — burny_tech · 2026-10-16
- Cornell and Amazon revisit the 16x pretraining compute gap: diffusion LMs need only 3-8x — LucaAmb · 2026-10-16
- AI trained on the fraction of human intelligence that fit the page — r0ck3t23 · 2026-10-16
- Quotient Recurrent Reasoners hit 97.9% on Sudoku-Extreme with 78% fewer loops — burny_tech · 2026-10-16