Deepfake Detection Papers Show Significant Real-World Drop
markjeffrey · x · 2026-07-17
BitMind released its deepfake detection paper and evaluation results: they point out that many detectors perform well on academic benchmarks but see AUC drops of 45–50% on real-world content, reflecting structural issues from evolving generative models.
They evaluated a frozen model uniformly across 19 public benchmarks without fine-tuning on individual datasets, achieving or surpassing the strongest results in multiple papers, including the best commercial detector on Deepfake-Eval-2024. The evaluation harness is public, and the production API serves the same evaluation snapshot for external verification.
More from Safety
- Researcher quits Anthropic, says OpenAI and Anthropic are racing to self-improving superintelligence — ShakeelHashim · 2026-09-11
- Why So Many AI Researchers Think the Machines Could Kill Everyone — wiredmagazine · 2026-09-11
- California creates standards for independent AI auditors to verify lab safety testing — VraserX · 2026-09-11
- a16z podcast: why 2-3 person startups are absent from policy debates — a16z Podcast · 2026-09-11
- Researcher questions AI safety eval firm, citing 'blatantly sloppy' security and monitoring — Kyrannio · 2026-09-11
- Class action accuses Anthropic of overselling Claude subscriptions with deceptive usage multipliers — The Decoder · 2026-09-11