Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination
Shuo Liang, Yixing Ma, Pengfei Zhou, Xingyan Chen, Zihan Mei, Manting Li, Feihan Chen, Zhiwen Wang, Bin Xu, Haotian Zhang, Jiajun Song, Shiya Su, Run Liu, Zhenghang Ni, Yifa Yu, Jintao Hong, Bolong Feng, Yifei Liu, Zirui Zhang, Jingxuan Zhang, Songlin Zhao, Yifan Bai, Kang Tan, Yizhe Liu, Junhao Du, Yongtao Ge, Zhaopan Xv, Xinyuan Zhang, Mengru Ma, Chunhua Shen, Wei Wang, Yang You, Zheng Zhu, Kaipeng Zhang, Wangbo Zhao
cs.CV, cs.AI
2026-08-14
RA-Bench pairs 1,830 real crisis clips with 16,056 AI-generated ones. No detector family generalizes, and social-media processing cuts fine-tuned fake recall from 46% to 1.4%.
Video generators can now fabricate plausible footage of disasters, accidents and other crisis events. When such a clip circulates, two defenses matter: human eyes and automated detectors. This paper asks whether either holds up in crisis settings.
Existing detection benchmarks draw from generic web video and mostly text-to-video generation. Crisis misinformation works differently. Fabricators commonly condition on a real first frame so the clip stays anchored to a real event, and the content passes through transcoding, downsampling and news badges before anyone sees it. Neither property is covered by prior benchmarks.
RA-Bench is built in five stages. 675 public videos of real crises and other socially consequential events are collected and organized into 10 risk categories and 44 subcategories. Scene-level segmentation yields 5,774 candidates; two rounds of review leave 2,426; duration bounding, unified H.264 encoding and deduplication produce 1,830 real anchors averaging 10.08 seconds.
The generation side mirrors the actual manipulation. Gemini captions each real clip, and the caption plus the real first frame go to four open-source and five closed-source generators, producing 16,056 clips. Generated clips receive the same H.264 encode and audio removal as the real ones, so detectors cannot lean on codec cues. Real anchors and generated clips form matched pairs, and results are reported as paired AUC: each real video competes directly against its own fabricated counterpart.
Evaluation runs along three axes: generalization of 19 detection methods across sources, how generation quality, conditioning and seeds move detectability, and human judgment plus a social dissemination simulation.
Traditional detectors fall hardest. Seven of them post public-reference AUCs of 67.6–98.6%, but source-level means on RA-Bench land at 43.9–57.3%; 26 of 63 detector–source pairs sit below 50% AUC, meaning generated clips score as less fake than their matched real anchors. Public rankings do not transfer: Spearman correlation with the RA-Bench ranking is 0.26, and UnivFD falls from second to sixth. At 5% FPR the seven-detector mean catches only 2.8–7.5% of generated videos.
Zero-shot multimodal models are prompt-fragile. Gemini-3.1-Pro leads with 63.4% BAcc under Binary but drops to 54.3 on Seedance2.0. Qwen3.5-0.8B swings from 19.7% to 100% fake recall when the prompt format changes. GPT-5.5's explicit verdict predicts nearly everything as Real.
Fine-tuned detectors lean on shortcuts. Skyra reaches 68.5–69.5% mean BAcc with its official timestamp prompt; swapping timestamps for frame indices drops it to 54.4–54.9%. BusterX++ holds 4.1–9.1% fake recall, behaving like an always-Real classifier.
| Setting | Metric | Result |
| 7 traditional detectors (public ref → closed-source mean) | mean AUC | 84.2% → 48.7% |
| Skyra-SFT (timestamps → frame indices) | mean BAcc | 68.5% → 54.4% |
| Fine-tuned detectors, before → after full dissemination stack | mean fake recall | 46.0% → 1.4% |
| Gemini Binary on HumanProof | BAcc | 54.7% (source-matched ref 61.2%) |
A 20-reviewer study identifies 68.6% of open-source generated clips but only 52.9% of closed-source ones, with Seedance2.0 at 40.7%. In the other direction, 22.8% of real-video judgments get labeled Generated. The 633 generated clips that all five reviewers called Real form the HumanProof subset, where Gemini is near chance under all three prompts and the traditional detectors average 47.5% AUC.
The LastMile simulation stacks four operations on 150 anchors: VP9-to-H.264 transcoding, half-resolution downsampling, 8 fps conversion, and a synthetic news badge. Under the full stack, mean fake recall across five fine-tuned configurations collapses from 46.0% to 1.4% as predictions slide toward Real. The badge alone cuts fake recall from 29.9% to 14.0%. Traditional detectors drop from 51.4% to 47.3% mean AUC with a rank correlation of 0.07 before versus after: ForgeLens falls from 61.6% to 35.6% while DeCoF rises from 59.9% to 62.3%.
The contribution is not a new detector but a decomposition of reliability into separately testable properties: cross-source generalization, prompt sensitivity, protocol dependence and class bias. For trust-and-safety teams the implication is blunt: no single detector currently justifies a verdict on crisis footage, and the paper argues for modular workflows that fuse evidence from multiple detectors.
On the generation side, the stability of detection patterns across seeds indicates difficulty is a generator property rather than sampling luck. The authors also suggest strong detectors can serve as reward models when post-training video generators.
The authors list three: the benchmark is a snapshot of nine generators and needs versioned updates; only the visual channel is tested although Seedance2.0 already generates synchronized audio; generation and dissemination are separate controlled stages, so multi-stage pipelines that combine generation with selective editing, captions and repeated platform processing are untested. Human judgments were collected in a controlled interface without real social context.
Two more concerns stand out. The fabrication recipe covers only first-frame image-to-video; audio forgery and partial editing are untouched. The four dissemination operations are synthetic, and how closely they match real platform processing is not verified.