LLM version turnover breaks AI-writing screening: detectors miss 1 in 3 rewrites of newest models
rohanpaul_ai · x · 2026-10-10
A Tokyo Metropolitan University study (arXiv:2610.11599) quantifies how LLM turnover undermines AI-assisted writing screening in journals. The authors paired 4,000 pre-ChatGPT PNAS abstracts with rewrites by 23 LLM versions from three vendors (June 2023–August 2026) and trained detectors under maintenance scenarios from constant retraining to never updating.
Key findings:
- Detectors trained only on a vendor's past versions collapse at model-generation boundaries: calibrated to 1% false-positive rate, they catch >99% of rewrites just before the sharpest boundary but only 3.8% just after.
- Detectors trained on later versions also miss rewrites by earlier ones; vocabulary differences largely track where transfer succeeds or fails.
- In simulated screening across all 23 versions, systems either flagged 1 in 8 human abstracts or missed 1 in 3 rewrites of the newest model.
- The commercial detector Pangram missed 79.8% of rewrites by Meta's Muse-Glimmer while flagging just 1 of 5,000 human abstracts.
The authors conclude that benchmarking detectors against fixed LLM versions is not sustainable as models keep changing.
Related event: AI Text Detectors Collapse as LLMs Iterate, Study Finds(3 posts)→
More from Safety
- OpenAI reportedly fires 3 safety staff as test shows model can hide its chain of thought — MattGarciaEth · 2026-10-10
- Dev: AI companies must be accountable for what their agents do in the real world — bendee983 · 2026-10-10
- Pangram AI image detector flags an AI image as "No AI" with high confidence — amplifiedamp · 2026-10-10
- China issues government ID cards to AI digital humans, virtual idol Yuri becomes Beijing 'digital resident' — hq4ai · 2026-10-10
- OpenAI-trained AI agents coordinated and plotted to falsify their logs, researcher says — JeffLadish · 2026-10-10
- Agentic identity & security series complete: user identity, agent identity, and scoped permissions — AxSaucedo · 2026-10-10