What is FakeShield
FakeShield is a multimodal large language model framework launched by researchers at Peking University, capable of detecting and locating image forgery. The framework can assess the authenticity of images, generate masks of tampered regions, and provide judgment evidence based on pixel-level and image-level tampering clues. FakeShield uses GPT-4o to enhance existing datasets, creating the Multimodal Tampering Description Dataset (MMTDSet), used for training tampering analysis capabilities. FakeShield contains two key modules: the Domain Tag-guided Explainable Forgery Detection Module (DTE-FDM) and the Multimodal Forgery Localization Module (MFLM), responsible for detection and localization tasks, respectively. FakeShield performs excellently in the detection and localization of various tampering techniques such as Photoshop, DeepFake, and AIGC editing, providing explainable solutions that outperform traditional methods.
FakeShield's main features
- Image authenticity assessment: Determines whether an image has been tampered with.
- Tampered region localization: Generates masks of tampered regions in images.
- Tampering clue analysis: Provides judgment evidence based on pixel-level and image-level tampering clues.
- Multimodal data processing: Combines visual and language models to improve detection accuracy and explainability.
FakeShield's technical principles
- Multimodal framework design: FakeShield is based on a multimodal large language model (M-LLM), integrating visual and textual information to improve detection and localization accuracy.
- Dataset enhancement: Uses GPT-4o to enhance the existing IFDL dataset, creating MMTDSet and providing richer training samples.
- Domain tag guidance: Introduces Domain Tags to distinguish different types of tampered data, enhancing the model's ability to recognize different tampering types.
- Explainability module: Develops the DTE-FDM module, providing detection evidence based on analyzing image features and generating detailed textual descriptions.
- Forgery localization module: Uses the MFLM module, combining visual language features to precisely locate tampered regions.
FakeShield's project address
- GitHub repository：https://github.com/zhipeixu/FakeShield
- arXiv technical paper：https://arxiv.org/pdf/2410.02761
FakeShield's application scenarios
- Social media content moderation: Automatically detects and filters tampered images on social platforms, preventing the spread of fake news and misleading content.
- Legal forensics: In court evidence collection, identifies whether image evidence has been tampered with, ensuring the authenticity and validity of evidence.
- News media: Helps news organizations verify the authenticity of news images and videos, maintaining the accuracy and credibility of news reports.
- Copyright protection: Provides copyright owners with tools to detect and locate unauthorized use or tampered images, protecting intellectual property.
- Security monitoring: In the field of security monitoring, ensures the authenticity of surveillance images, preventing fraud or illegal acts using tampered images.