Purpose
DataTrove is a library for processing, filtering, and deduplicating text data at large scale. It frees text data processing from the chaos of scripts, providing a set of platform-independent, customizable pipeline processing modules. Its processing pipeline can be run directly locally or on a Slurm cluster. Its low memory usage and multi-step design make it suitable for large-scale workloads, such as processing training data for large language models.
Main capabilities
DataTrove provides a set of preset commonly used processing modules and provides a framework that makes it easy to extend with custom functionality. Each module in the pipeline processes data in the datatrove Document format. A Document contains: text (the actual text content of each sample), id (the unique string ID of that sample), metadata (a dictionary that can hold any additional information). Each pipeline module receives a Document generator as input and returns another Document generator.
Pipeline module types include: readers (read data from different formats and generate Documents), writers (save Documents to disk or the cloud in different formats), extractors (extract text content from raw formats such as web page HTML), filters (remove some Documents according to specific rules or conditions), stats (collect dataset statistics), tokens (tokenize data or count tokens), dedup (deduplication).
Usage
Installation requires Python 3.10+. Different features can be combined by repeating --extra, for example uv sync --extra processing --extra s3. Available combinations include: all installs everything; io provides dependencies for reading warc/arc/wet files and arrow/parquet, Optimized-parquet formats; processing is used for text extraction, filtering, and tokenization; s3 supports s3; cli provides command-line tools; ray provides a distributed computing engine; inference is used for LLM inference pipelines; decont is used for decontamination in combination with lighteval; multilingual is used for multilingual text processing. For example, to install all: uv sync --extra all.
A pipeline isList of processing steps to be executed (reading data, filtering, writing to disk, etc.); an executor runs a specific pipeline on a given execution environment (slurm, multi-CPU machine, etc.); a job is the execution of a pipeline on a given executor; a task is a component of a job, used for parallel execution, typically having each task process one data shard, and DataTrove records which tasks have completed so that on restart only the unfinished tasks are run; a file is a single input file (.json, .csv, etc.), each file is processed by a single task, and DataTrove does not automatically split one file into multiple parts, so to make full use of parallelism you should use multiple medium-sized files rather than a single large file; a shard is a set of input data (usually a set of files) that is assigned to a specific task, with each task processing a different, non-overlapping data shard; a worker is a computing resource that executes one task at a time, for example with 50 CPU cores you can run a LocalPipelineExecutor with workers=50 and execute 50 tasks simultaneously.
The number of tasks determines the degree of parallelism and the time consumed by each processing unit. When the number of tasks is small (each task processes many files), if a failure occurs the whole thing must be rerun from the beginning; when the number of tasks is large and each is smaller, rerunning a single failed task takes less time. If tasks is greater than files, some tasks will not process any data, so it is usually unnecessary to set tasks greater than files.
The executors include: LocalPipelineExecutor (launches the pipeline on the local machine, with options tasks for the total number of tasks, workers for the number of tasks running simultaneously, and start_method for the method of creating the multiprocessing Pool); SlurmPipelineExecutor (launches the pipeline on a slurm cluster, using slurm job arrays to manage and group tasks); RayPipelineExecutor (launches the pipeline on a ray cluster, using ray tasks for parallel execution); JobsPipelineExecutor (experimental, may change or be removed without notice, runs the pipeline on Hugging Face Jobs).
Calling the executor's run method executes its pipeline. DataTrove records which tasks have successfully completed by creating markers (empty files) in the ${logging_dir}/completions folder. If some tasks fail after the job ends, you can restart the exact same executor, and DataTrove will check and run only the previously unfinished tasks. If you restart the pipeline because some tasks failed, do not change the total number of tasks, otherwise it will affect the shard distribution of the input files.
Regarding paths, DataTrove supports a variety of input/output sources through fsspec. The ways to provide paths to the datatrove module (such as the input_folder, logging_dir, data_folder and other parameters) are: str (the simplest, passing a single string, for example /home/user/mydir, s3://mybucket/myinputdata, hf://buckets/myorg/my-bucket/raw/, hf://datasets/allenai/c4/en/); (str, fsspec filesystem instance) (a string path and a fully initialized filesystem object); (str, dict) (a string path and a dictionary of options for initializing the fs); DataFolder (directly initialize a DataFolder object and pass it in). These parameter combinations are parsed by get_datafolder.
Log messages support coloring. By default, console messages automatically detect coloring, and log files (logs/task_XXXXX.log) have coloring disabled. Coloring can be explicitly enabled or disabled through environment variables: setting DATATROVE_COLORIZE_LOGS to "1" adds ANSI colors to console log messages, and setting it to "0" disables coloring; setting DATATROVE_COLORIZE_LOG_FILES to "1" adds ANSI colors to log messages saved to logs/task_XXXXX.log.
Examples and Documentation Topics
The examples listed in the materials include: fineweb.py (a complete reproduction of the FineWeb dataset); process_common_crawl_dump.py (reads commoncrawl warc filesrun method to execute its pipeline. DataTrove records which tasks have successfully completed by creating markers (empty files) in the ${logging_dir}/completions folder. If some tasks fail after the job ends, you can restart the exact same executor, and DataTrove will check and run only the previously unfinished tasks. If you restart the pipeline because some tasks failed, do not change the total number of tasks, otherwise it will affect the shard distribution of the input files.
Regarding paths, DataTrove supports a variety of input/output sources through fsspec. The ways to provide paths to the datatrove module (such as the input_folder, logging_dir, data_folder and other parameters) are: str (the simplest, passing a single string, for example /home/user/mydir, s3://mybucket/myinputdata, hf://buckets/myorg/my-bucket/raw/, hf://datasets/allenai/c4/en/); (str, fsspec filesystem instance) (a string path and a fully initialized filesystem object); (str, dict) (a string path and a dictionary of options for initializing the fs); DataFolder (directly initialize a DataFolder object and pass it in). These parameter combinations are parsed by get_datafolder.
Log messages support coloring. By default, console messages automatically detect coloring, and log files (logs/task_XXXXX.log) have coloring disabled. Coloring can be explicitly enabled or disabled through environment variables: setting DATATROVE_COLORIZE_LOGS to "1" adds ANSI colors to console log messages, and setting it to "0" disables coloring; setting DATATROVE_COLORIZE_LOG_FILES to "1" adds ANSI colors to log messages saved to logs/task_XXXXX.log.
Examples and Documentation Topics
The examples listed in the materials include: fineweb.py (a complete reproduction of the FineWeb dataset); process_common_crawl_dump.py (reads commoncrawl warc files, extract text content, filter it, and save the results to s3; runs on slurm); tokenize_c4.py (reads data directly from the huggingface hub, uses the gpt2 tokenizer to tokenize the English portion of the C4 dataset); estimate_tokens.py (estimates the total token count of large HF datasets); smol_data.py (builds approximately 100B token subsets, 50-30-20 mixtures, and shuffled variants for multiple large Hugging Face datasets); minhash_deduplication.py (a complete pipeline for running minhash deduplication on text data); sentence_deduplication.py (runs sentence-level exact deduplication); exact_substrings.py (runs ExactSubstr, requires specifying a repository); finephrase.py (standalone examples for generating synthetic datasets at scale with multiple prompt templates).
The documentation topics listed in the materials include: quick start examples, DataTrove Document, pipeline module types, complete pipelines, executors (LocalPipelineExecutor, SlurmPipelineExecutor, RayPipelineExecutor, JobsPipelineExecutor), DataFolder / paths, practical guides (reading data, synthetic data generation, custom rollouts, ready-to-use generation scripts, advanced configuration, progress monitoring, benchmarking, extracting text, filtering data, saving data, deduplicating data, summary statistics), custom modules (simple data, custom functions, custom modules).
Applicable scenarios and limitations
The applicable scenarios are mainly large-scale text data processing, filtering, and deduplication, such as processing large language model training data, reading and extracting commoncrawl data, tokenizing datasets or estimating token counts, generating synthetic datasets, etc. In terms of limitations: Python 3.10+ is required; when processing large-scale tasks, it is important to set the number of tasks appropriately; if restarting due to failure, the total number of tasks should not be changed; each file is processed by a single task and will not be automatically split, so to fully utilize parallelism, multiple medium-sized files must be used; JobsPipelineExecutor is an experimental feature and may be changed or removed. Other unspecifiedThe information described in the materials is not yet provided.
