What is WebLI-100B
WebLI-100B is an ultra-large-scale dataset containing 100 billion image-text pairs, launched by Google DeepMind, used for pretraining vision-language models (VLMs). WebLI-100B is an expanded version of the WebLI dataset, built by collecting a large number of images and their corresponding captions or page titles from the web as text pairing information. The scale of WebLI-100B is ten times that of the previous largest vision-language dataset, using massive data to enhance the model's understanding of long-tail concepts, cultural diversity, and multilingual content. During construction, the researchers only performed basic data filtering to preserve as much language and cultural diversity as possible. The emergence of WebLI-100B provides an important foundational resource for training more inclusive multimodal models.
Main Features of WebLI-100B
- Supports Large-Scale Pretraining: WebLI-100B provides 100 billion image-text pairs, offering rich data resources for the pretraining of VLMs and significantly improving model performance on various tasks.
- Enhances Cultural Diversity: It contains images and texts from different cultural backgrounds, helping models better understand and generate visual and linguistic content related to different cultures.
- Strengthens Multilingual Capabilities: The dataset contains text in multiple languages, which helps improve model performance on low-resource languages and promotes the development and application of multilingual tasks.
- Supports Multimodal Tasks: The data in WebLI-100B is used for various multimodal tasks, such as image classification, image caption generation, visual question answering, etc., providing strong support for the development of multimodal models.
Technical Principles of WebLI-100B
- Data Collection：
- Source: The data in WebLI-100B mainly comes from the internet, collecting images and their corresponding text descriptions (such as image alt text or page titles) through large-scale web crawling.
- Scale: The dataset contains 100 billion image-text pairs, making it one of the largest vision-language datasets to date.
- Data Filtering：
- Basic Filtering: To ensure data quality and diversity, WebLI-100B only performed basic data filtering, such as removing harmful images and personally identifiable information (PII), to preserve as much language and cultural diversity as possible.
- Quality Filtering (Optional): The study also explored using models such as CLIP for data filtering to improve data quality, but this filtering may reduce the representation of certain cultural backgrounds.
- Data Processing：
- Text Processing: The alt text and page titles of images are used as paired text, and a multilingual mt5 tokenizer is used for tokenization to ensure the diversity and consistency of text data.
- Image Processing: Images are resized to a resolution of 224×224 pixels to meet the model's input requirements.
WebLI-100B Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2502.07617
Application Scenarios of WebLI-100B
- AI Researchers: Used for model pretraining, exploring new algorithms, and improving vision-language model performance.
- Engineers: Develop multilingual and cross-cultural applications, such as image captioning, visual question answering, and content recommendation systems.
- Content Creators: Generate multilingual image descriptions and tags to enhance content localization and diversity.
- Cross-cultural researchers: Analyze images and texts from different cultural backgrounds to study cultural differences.
- Educators and students: As a teaching resource, learn multimodal data processing and analysis.