What Is Nemotron-CC
Nemotron-CC is a large-scale pretraining dataset introduced by the NVIDIA team that can transform Common Crawl data into high-quality corpora suitable for long-sequence pretraining. The Nemotron-CC dataset combines methods such as classifier ensembles, synthetic data rephrasing, and reduced reliance on heuristic filters to achieve a better balance between data volume and quality. Nemotron-CC contains 6.3 trillion tokens, of which 4.4 trillion are globally deduplicated original tokens and 1.9 trillion are synthetically generated tokens. Nemotron-CC demonstrates superiority in both short-term (1T tokens) and long-term (15T tokens) training, especially on tasks such as MMLU, where compared with DCLM and the Llama 3.1 model, Nemotron-CC significantly improves model accuracy. The development of the Nemotron-CC dataset provides richer and more diverse data resources for the training of large language models.
Main Features of Nemotron-CC
- Provides a high-quality pretraining dataset: Nemotron-CC provides a high-quality, large-scale pretraining dataset for large language models (LLMs), especially suitable for long-sequence training, such as training needs of 15T tokens.
- Balances data volume and quality: Without significantly sacrificing data quality, it greatly increases the scale of the dataset, contains more unique real tokens, and improves model performance in long-sequence training.
- Supports model performance improvement: Experiments show that models trained on Nemotron-CC perform excellently on multiple benchmark tasks, especially on tasks such as MMLU, and can significantly improve model accuracy compared with other existing datasets.
Technical Principles of Nemotron-CC
- Optimized text extraction: Justext is chosen as the HTML-to-text extractor because it performs better in extracting high-quality tokens, effectively improving the initial quality of the dataset.
- Model-based quality labeling：
- Classifier ensemble: Three different quality classifiers are built, each with different high-quality preferences. Based on the ensemble of the three classifiers, all documents are scored, and the crawled corpus is divided into different quality levels according to quality scores.
- Quality label assignment: Fine-grained quality scores are further clustered into 5 downstream task quality categories, and based on continuous pretraining and task performance evaluation, a quality label more consistent with actual performance is assigned to each category.
- Synthetic data generation：
- Rephrasing of low-quality data: For low-quality documents, rephrasing is used to reduce noise and errors while retaining useful information. Low-quality documents are rephrased with Wikipedia-style prompts, effectively reducing errors and redundancy and improving the level of formatting.
- Expansion of high-quality data: For high-quality documents, methods such as generating diverse question-answer pairs, refinement, knowledge extraction, and knowledge lists are used to obtain more unique tokens and enrich the content and diversity of the dataset.
- Dataset integration：
- Large-scale data synthesis: The Mistral NeMo 12B model is used to generate more than 1.8 trillion synthetic tokens, including 336.3 billion tokens generated from low-quality documents and 1.5 trillion tokens generated from high-quality documents.
- Dataset construction: The above techniques and methods are applied to 99 snapshots of Common Crawl (CC-MAIN-2013-20 to CC-MAIN-2024-30) to construct a dataset of 6.3 trillion tokens, of which 4.4 trillion are globally deduplicated original tokens and 1.9 trillion are synthetically generated tokens.
Nemotron-CC project address
- Project official website：https://developer.nvidia.com/blog/announcing-nemotron-cc
- arXiv technical paper：https://arxiv.org/pdf/2412.02595
Application scenarios of Nemotron-CC
- Pretraining large language models: Suitable for long-sequence training, such as 15T tokens, improving model performance on complex tasks.
- Fine-tuning and task-specific adaptation: Facilitates rapid adaptation of models in multi-task learning and domain-specific tasks, improving task performance.
- Text generation tasks: Used to generate high-quality text, such as news and stories, and to improve the naturalness of dialogue systems.
- Research and development: Helps explore improvements in model architecture and training methods, and provides benchmark testing resources.
- Education and training: Generates educational resources, assists language learning, and enhances the richness of educational content.