TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection
Huiyuan Liu, Zhiming Ma, Yanxing Liu, Shun Zhang, Qifan Wang, Di Liu, Yifan Wang, Yuyang Deng, Haoyang Meng, Yijin Zhou, Yuxi Zhao, Chengxian Hu, Peidong Wang, Peng Chen
cs.SD, cs.CL
2026-09-16
TeleAntiFraud 2.0 freezes 900 Chinese calls a month; near-domain siblings drop text Macro-F1 from 1.00 to 0.65-0.68 and collapse many audio models to all-FRAUD.
Telecom-fraud scripts change quickly, and they are written to sound like ordinary service, risk-control, or verification calls. Two evaluation failures follow. A frozen test set cannot absorb scam patterns that appear after release. If the negatives come from unrelated topics, a model can separate fraud from normal speech by topic words alone, and the score looks stronger than the decision really is.
TeleAntiFraud-28k already framed spoken telecom fraud as an audio-text benchmark, but the authors note that it does not release raw transcripts, that audio-path prefixes correlate with labels, and that it lacks enough lawful near-domain counterparts. The missing object is a contract that can ingest new cases every month without rewriting yesterday's test set.
The construction stack is a Mixed-Tree Anti-Fraud Generation Pipeline plus monthly frozen snapshots.
Online fraud-case abstracts become scenario profiles: participants, persuasion strategy, risk nodes. A mixed tree then expands under that shared profile and opening, with depth cap 4 and at most 3 plot actions per node. Sibling leaves keep the same people, setting, and early risk language, and split only on later actions. One path sends the receiver to official verification; the other asks for a remote card photo and piles on freeze threats. The label tracks the finished trajectory, not the topic.
Six agents realize the dialogue: caller and receiver speakers, branch and stop controllers, and two delivery-state agents. Higgs TTS 3 renders role-matched speech. Each monthly snapshot freezes 900 Chinese calls, 600 fraud against 300 near-domain non-fraud, together with audio, label rationales, prompts, model responses, and provenance. June/V1 and July/V2 are the two released sets.
The text-side ablation is the cleanest evidence. Moving negatives from unrelated calls and ordinary telecom service to mixed-tree siblings raises lexical overlap from 0.005 and 0.161 to 0.274. Logistic regression, SVM, and RoBERTa fall from perfect Macro-F1 to 0.680, 0.673, and 0.650.
The same TF-IDF+SVM stack scores 0.9950 Macro-F1 on TeleAntiFraud-28k ASR text, 0.9286 on TeleAntiFraud 2.0 dialogue text, and 0.7998 on the 2.0 ASR-test split, where non-fraud recall is 0.5800 and the predicted-fraud ratio is 0.7878.
On the full 900-call snapshots, 11 of 27 V1 configurations and 15 of 27 deduplicated V2 configurations show an all-FRAUD signature: fraud recall near 1.0, accuracy stuck to the 2:1 prior, fraud F1 stuck near 0.80. Gemini-Pro audio on June is 0.790/0.654/0.980, not fully collapsed. Qwen3-Omni, DeepSeek-V3, and GPT-4o ASR runs often print 0.800/0.667/1.000. Resampling the prior to 1:2 drops the all-FRAUD baseline's fraud F1 from 0.80 to 0.50 while Balanced Accuracy stays at 0.50.
Construction audits: BGE distances on 813 dialogues are 0.0188 within tree and 0.3351 across trees. Ten experts on 1000 item judgments agree with corrected gold at 0.792 on average.
The benchmark separates "can the model hear risk vocabulary" from "can it wait for the action that actually decides the label." If an audio anti-fraud paper still reports fraud F1 against unrelated negatives, that number carries almost no information. The reproducible protocol is frozen snapshots, Macro-F1 plus both recalls, and stored raw predictions.
This is a synthetic diagnostic instrument. It does not claim to match the distribution of real phone calls. For a deployed detector, the closer question is whether lawful risk-control calls get killed as fraud.
The authors say the dialogues and TTS cannot stand in for real call traffic; two monthly snapshots are too few for long-horizon drift claims; the 35 reference voices are 29 male and 6 female; dialogue quality is a single-annotator 80-item pilot scoring about 4.1-4.6 on a 1-5 scale. Expert agreement ranges 0.70-0.88, so hard cases remain. Many full-set models collapse to all-FRAUD, and the paper does not name a stable leader. If the generator is later used for training, leave-one-tree-out F1 drops from 0.883 in-domain to 0.605, which is a real leakage risk.