Testing Text Detectors With Pre-Internet Books
cephaloform · x · 2026-07-17
- This retweet discusses how, when evaluating text detectors like Pangram, you can't just test them on public texts from before the ChatGPT era, or the results will be contaminated by the fact that "the model has seen the training data."
- To verify it isn't just memorizing old text, someone found old books that don't exist on the internet, scanned them, and ran tests totaling nearly 3 million words.
- In the results, the detector still found a few traces of AI: 3 sections suspected of AI assistance, and 1 section suspected of AI generation.
- The key takeaway isn't the conclusion itself, but the rigorous anti-"data leakage" evaluation approach it provides.
More from Research
- PNAS paper shows a tiny billiard-ball system is a universal computer — undecidability lives in two dimensions — eigensteve · 2026-09-11
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11
- LoMa Paper Ships REALLY HardPairs Dataset, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Johns Hopkins Launches Full-Stack Hands-on Robot Learning Class with SO-101 Arm Kits — _krishna_murthy · 2026-09-11
- SyncWorld: In-Context Robot World Model Simulates Unseen Views and Embodiments Zero-Shot — ChongZzZhang · 2026-09-11
- A 3D Pose Dataset for Dogs Released — ducha_aiki · 2026-09-11