What is SafeEar
SafeEar is an AI audio forgery detection framework jointly developed by Zhejiang University and Tsinghua University, which protects user privacy while detecting audio forgeries. It employs a disentangled model based on a neural audio codec to separate the acoustic information and semantic information of speech, and uses the acoustic information for detection, effectively preventing privacy leakage. SafeEar performs excellently on multiple benchmark datasets, with an Equal Error Rate (EER) as low as 2.02%, and can resist content recovery attacks. SafeEar provides multilingual support and has built the CVoiceFake dataset containing 1.5 million multilingual audio data entries, providing valuable resources for speech forgery detection research.
Main Features of SafeEar
- Privacy-Preserving Deepfake Detection: By separating the semantic and acoustic information of speech, SafeEar uses only acoustic information to detect deepfake audio, protecting the privacy of speech content during the detection process.
- Multilingual Support: SafeEar can process and detect audio data in multiple languages, including but not limited to English, Chinese, German, French, and Italian.
- Efficient Forgery Detection: Tests on multiple public benchmark datasets show that SafeEar's Equal Error Rate (EER) is as low as 2.02%, demonstrating efficient detection capability.
- Anti-Content Recovery Technology: SafeEar combines real-world scenario-based codec enhancement and anti-content recovery technology, maintaining high detection accuracy even under adversarial attacks.
- Real-Environment Enhancement: By simulating the diversity of audio channels in real environments, SafeEar enhances the model's generalization ability across different communication scenarios.
- Open-Source Resources: SafeEar provides open access to its paper, code, and dataset, promoting further research and application development in the research community.
- Dataset Construction: SafeEar has built the CVoiceFake dataset, containing 1.5 million multilingual audio samples, providing a standardized test benchmark for speech forgery detection.
Technical Principles of SafeEar
- Semantic-Acoustic Information Separation: SafeEar uses a neural audio codec model to disentangle the semantic information (such as language content) and acoustic information (such as timbre, intonation, and rhythm) in audio. This separation ensures that the specific content of the audio is not leaked when detecting forged audio, protecting user privacy.
- Acoustic Information Analysis: The system analyzes only the acoustic features of the audio, not the semantic content. By focusing on acoustic features such as pitch, timbre, and rhythm, SafeEar can effectively identify forged audio.
- Multilingual Support: SafeEar can process audio data in multiple languages, employing desemanticization processing to ensure that specific semantic content is not exposed during analysis.
- Anti-Content Recovery Technology: SafeEar combines real-world scenario-based codec enhancement and anti-content recovery technology to resist various audio deepfake methods, ensuring high detection accuracy even under adversarial attacks.
- Transformer Classifier: SafeEar employs a Transformer classifier based on acoustic input. The classifier has shown potential in forgery detection and can improve the accuracy and efficiency of detection.
SafeEar Project Address
- Project Homepage：safeearweb.github.io/Project/
- Github Repository：https://github.com/LetterLiGo/SafeEar
- Technical Paper：https://safeearweb.github.io/Project/files/SafeEar_CCS2024.pdf
Application Scenarios of SafeEar
- Social Media and Public Forums: On these platforms, the forgery of audio content may be used to mislead or deceive, and SafeEar can be used to detect and flag such forged content.
- Legal and Judicial Systems: In legal proceedings, the authenticity of audio evidence is crucial. SafeEar can help verify the authenticity of recordings.
- Financial Institutions: In customer service and transaction verification, speech recognition systems may be threatened by forged audio. SafeEar can improve transaction security.
- Government and Security Agencies: In the fields of national security and public safety, detecting forged audio information is crucial. SafeEar can help identify potential threats and false information.
- Online Education: In online courses and exams, SafeEar can be used to ensure the authenticity of audio materials and prevent academic fraud.