What is Crawl4LLM
Crawl4LLM is an intelligent crawler system jointly open-sourced by Tsinghua University and Carnegie Mellon University, improving the pre-training efficiency of large language models (LLMs). Crawl4LLM is based on intelligently evaluating the value of web pages for LLM pre-training, prioritizing the crawling of high-value web pages, and is nearly 5 times more efficient than traditional crawlers. Crawl4LLM supports three crawling modes: intelligent mode, random crawling mode, and link-count-based crawling mode. It also features periodic saving of crawler state, data visualization, and can seamlessly integrate with the DCLM framework, directly used for model training.
Main Features of Crawl4LLM
- Intelligent web page selection: The system is based on evaluating the value of web pages for LLM pre-training, prioritizing the crawling of high-value web pages, improving data quality and reducing the crawling of invalid data.
- Multiple crawling modes：
- Intelligent mode: Based on web page value evaluation, prioritizing the crawling of high-value web pages.
- Random mode: Randomly crawls web pages, suitable for scenarios without precise requirements.
- Link-count-based mode: Crawls based on the number of web page links, suitable for large-scale data collection.
- Periodic saving of crawler state: Supports periodic saving of crawler state, allowing crawling to continue from the interruption point even if interrupted, avoiding data loss.
- Data browsing and visualization: Provides data browsing tools and a visualization interface, making it convenient for users to monitor crawling progress and results in real time.
- Seamless integration with the DCLM framework: The crawled data is used for LLM pre-training, improving data flow efficiency and accuracy.
Technical Principles of Crawl4LLM
- Pre-training influence scoring: Crawl4LLM uses a pre-training influence scorer (such as DCLM fastText) to score web pages. The scorer evaluates the contribution of web pages to LLM pre-training based on indicators such as content quality and relevance. In each crawling iteration, newly discovered web pages are scored by the scorer, and the crawling priority is determined based on the score.
- Priority queue: Web pages are sorted based on a priority queue, prioritizing the crawling of the highest-scoring web pages, replacing the scheduling mechanism of traditional crawlers based on graph connectivity (such as PageRank). Based on the priority queue, Crawl4LLM quickly discovers and crawls the most valuable web pages for pre-training, reducing the crawling of low-value web pages.
- Multi-dimensional data evaluation: Crawl4LLM considers the quality of web page content and combines multi-dimensional indicators such as the number of links and content length for comprehensive scoring. It analyzes the link relationships of high-scoring web pages to discover more potential high-value web pages.
- Simulation and optimization: Large-scale simulation experiments are conducted on the ClueWeb22 dataset to verify effectiveness in different scenarios. Based on the experiments, algorithm parameters are optimized to ensure the best pre-training effect under a limited crawling volume.
- Reducing burden on websites: Reduces unnecessary web page crawling, lowers the traffic burden on websites, and improves the compliance of crawling behavior. Crawl4LLM reduces the pressure of data crawling on websites and network resources, promoting a more sustainable way of acquiring pre-training data.
Project Address of Crawl4LLM
- GitHub repository：https://github.com/cxcscmu/Crawl4LLM
- arXiv technical paper：https://arxiv.org/pdf/2502.13347
Application Scenarios of Crawl4LLM
- LLM pre-training data collection: Efficiently obtain high-quality data for pre-training large language models.
- Search engine optimization: Improve search result quality and optimize user experience.
- Dataset construction: Quickly filter and build high-quality corpora to meet research and commercial needs.
- Network monitoring and analysis: Monitor network dynamics and analyze hot topics and information dissemination.
- Enterprise-level data collection: Precisely capture data in specific domains for knowledge management or market analysis.