What is ProX
ProX (Programming Every Example) is a framework designed to improve the quality of pretraining data for large language models. Unlike traditional methods that rely on human experts to formulate rules, ProX treats data cleaning tasks as programming problems and supports models in automatically performing fine-grained operations such as string normalization and noise line removal. Small models (e.g., with 0.3B parameters) can also demonstrate data processing capabilities comparable to human experts. Experimental results show that when data processed by ProX is used for model pretraining, performance improvements of over 2% are achieved across various downstream tasks. Another notable advantage of ProX is its broad applicability across different model sizes and pretraining corpora, including in continued pretraining in specific domains (such as mathematics), where it can significantly improve model performance without domain-specific design. ProX saves training FLOPs and provides a promising path for efficient pretraining of large language models.
Main Features of ProX
- Data Refinement: ProX refines large-scale datasets based on generating and executing programs, improving data quality for use in pretraining large language models.
- Automated Processing: Automatically performs fine-grained cleaning and improvement on each data sample without the intervention of human experts.
- Performance Improvement: Models pretrained on data processed by ProX show performance improvements of over 2% across multiple downstream tasks.
- Domain Flexibility: Applicable to different domains, including mathematics, improving accuracy without the need for domain-specific design.
- Resource Savings: Compared with data synthesis methods based on large language models, ProX significantly reduces computational resource requirements while maintaining results.
Technical Principles of ProX
- Model Adaptation: ProX first fine-tunes a small base language model on seed data to adapt it to the data refinement task.
- Program Generation: The adapted model generates a data processing program for each sample in the pretraining corpus; the program includes operations such as filtering, string normalization, and removal of noise lines.
- Program Execution: The generated programs are executed by a predefined executor, producing a refined corpus ready for pretraining.
- Two-Stage Refinement: ProX includes two stages, document-level programming and chunk-level programming, performing coarse-grained and fine-grained data refinement respectively.
- Function Calling: ProX enhances data quality based on flexible function calling, unified into specific transformation or cleaning processes.
- Computational Efficiency: ProX demonstrates how, with fewer pretraining computation FLOPs, investing additional computational resources in refining the pretraining corpus can enable more efficient pretraining.
ProX Project Address
- Project Official Website：gair-nlp.github.io/ProX
- GitHub Repository：https://github.com/GAIR-NLP/ProX
- HuggingFace Model Library：https://huggingface.co/gair-prox
- arXiv Technical Paper：https://arxiv.org/pdf/2409.17115
Application Scenarios of ProX
- Large Language Model Pretraining: ProX can improve the dataset quality for pretraining large language models on large amounts of high-quality text data.
- Data Cleaning and Preprocessing: In data mining, natural language processing, and other machine learning tasks, ProX automatically performs data cleaning and preprocessing steps, reducing manual intervention.
- Domain Adaptation Training: For domain-specific applications, such as healthcare, law, or finance, ProX can optimize datasets to better adapt to specialized terminology and context.
- Continual Learning: In continual learning or incremental learning scenarios, ProX helps models adapt to new information and changes by continuously refining data.
- Data Synthesis: ProX can generate high-quality synthetic data to enhance existing datasets, especially in data-scarce domains.