Brief
Telecom fraud benchmark swaps easy negatives for near-domain calls, and classifier scores fall to 0.65-0.68 Macro-F1
TeleAntiFraud 2.0 is an audio-based telecom fraud detection benchmark built from a Mixed-Tree Anti-Fraud Generation Pipeline and evaluated under a monthly frozen protocol, in which each frozen set holds 900 Chinese calls. Its authors report that near-domain negative cases, rather than topic-separated ones, are what separates strong headline scores from realistic ones.
The change is in the negatives and in the update mechanism. Each monthly frozen set contains 900 Chinese calls - 600 fraud and 300 near-domain non-fraud - with audio, labels, prompts, manifests and provenance records frozen together, so newly observed scam patterns can be added without overwriting previously established test sets. The pipeline turns online fraud-case abstracts into profile-grounded scenarios, expands them through mixed-tree generation, builds fraud and non-fraud dialogue paths under shared contexts, and renders validated dialogues as role-matched speech.
The consequence is a measurement, not a leaderboard. In controlled text experiments the source reports that three classifiers reach perfect macro-averaged F1 against unrelated or ordinary negatives, but drop to 0.65-0.68 once near-domain sibling negatives are used. Full-set audio and ASR+LLM evaluations further reveal class-prior shortcuts, prediction collapse and sensitivity to which snapshot is used. The artifact accompanying the work includes construction code, evaluation scripts, manifests and documentation.
Our reading
For anyone running a voice fraud model against call audio, this says the number you report is largely a function of how the negatives were chosen, and a topic-separated test set will flatter almost any classifier. Teams building telecom or contact-centre fraud detection, and anyone pairing ASR with an LLM on call audio, should treat near-domain negatives and per-snapshot reporting as a minimum ba…
What to do or watch
Rebuild one internal fraud detection test set so its negatives are near-domain calls under shared contexts rather than a different topic, then re-run your current classifier and report Macro-F1 per monthly snapshot separately; if the score collapses between snapshots, that sensitivity - not the peak score - is the number to fix before deployment.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- Each frozen set contains 900 Chinese calls, comprising 600 fraud and 300 near-domain non-fraud cases.
- Three classifiers achieve perfect macro-averaged F1 when evaluated against unrelated or ordinary negatives, but drop to 0.65-0.68 with near-domain sibling negatives.
- Evaluation uses a monthly frozen evaluation protocol that freezes the audio, labels, prompts, manifests and provenance records for each monthly evaluation set.
- Full-set audio and ASR+LLM evaluations reveal class-prior shortcuts, prediction collapse, and snapshot sensitivity.
- The accompanying research artifact includes the construction code, evaluation scripts, manifests, and documentation.
Sources
- arXivText stored 17 September 2026
How this story was checked. Written from the 1 page listed above, stored 17 September 2026; claims checked against that stored text on 17 September 2026.
What that means
- 5 of 5 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.