Most open-source AI text detectors can't hold a 0.5% false-positive rate
grumpyp2 · reddit · 2026-09-02
A team benchmarked every notable open-source AI-text detector under one protocol—thresholds matched to 0.5% FPR on 6,930 human docs, then recall measured on raw AI, humanizer-paraphrased, and frontier-model text (GPT-5.x, Claude, Gemini 3.x):
- 4 of 6 models can't reach 0.5% FPR at all; MAGE scores >0.9999 on 26% of ordinary human web text; the old OpenAI RoBERTa detector lands at AUC 0.31—worse than a coin flip.
- Humanizer-paraphrased text collapses everything: best model catches 42%, second best 4%.
- Every model over-flags non-native TOEFL essays—a systematic failure of the whole category.
Top scorer tropa-mini (ROC-AUC 0.968) is the authors' own, released as open weights (Apache-2.0) with full data and methodology for reproduction.
More from Models
- User finds GPT 5.6 Sol medium barely worse than high, and faster — iamsahaj_xyz · 2026-09-02
- Fable 5.1 beats Fable 5, matches Opus 5 on ML bench as refusals drop to 0/12 — xeophon · 2026-09-02
- Anthropic's Fable 5.1 claimed 45% savings, but Max users burn limits in under an hour — heypearlai · 2026-09-02
- Fable 5.1 drops and users are already one-shotting entire games in under 24 hours — eyishazyer · 2026-09-02
- OpenAI Astra safety data: more capable model, zero misaligned cyber attacks vs Sol's 56% — VoidStateKate · 2026-09-02
- Mid-Training Distillation Boosts Reasoning But Small Models Lose Fact Recall First — roydanroy · 2026-09-02