Audit: all 11 AI humanizer tools alter facts in 35-65% of documents with checkable details

Ziyuan Wang · hf · 2026-11-13

An audit of AI-text "humanizers" — tools that rewrite machine-generated text to evade detectors — checked whether checkable facts survive:

Method: a rule-based checker (validated by two AI annotators, precision 0.84-1.00, recall 0.93-0.95) verified numbers, dates, person/organization names, and quoted spans across rewrites.
Results: in the October 2026 leaderboard cycle, all 11 commercial tools changed or dropped at least one unit in 35-65% of documents containing one; counting only numbers and names, 10 of 11 tools still altered at least one in ≥20% of documents, mostly quoted spans. September was similar (19-50%).
An open 12B humanizer kept ≥97% of numbers/names on fact-dense federal text but reworded 67-78% of quoted spans against its own prompt. Unit retention and detector evasion showed no clear relation (Spearman rho=-0.46).
Detection side: a general-purpose decision model (Jev) with one zero-shot question separated human originals from humanized rewrites at AUROC 0.864, 0.18-0.22 above Binoculars, Fast-DetectGPT, and RADAR; but perplexity-based detectors scored below chance (0.37/0.39) on 2026-model AI-styled rewrites.
Hypotheses and analyses were preregistered; the checker and per-item results are released.

More from Safety