Chinese-Tiny-LLM

Purpose

The Chinese-Tiny-LLM repository contains the Chinese web corpus cleaning and pretraining code of the CT-LLM project, intended for researchers and developers who need to build Chinese large language models.

Main Capabilities

• CT-LLM: A 2B-parameter language model trained from scratch, mainly using Chinese data. The training corpus totals 1.2 trillion tokens, including 800 billion Chinese, 300 billion English, and 100 billion code tokens, combined with alignment techniques to improve Chinese processing capabilities. It shows good Chinese task performance on CHC-Bench and also has English capabilities through SFT.
• MAP-CC: An open-source Chinese pretraining dataset with a scale of 800 billion tokens, accompanied by a detailed pipeline for cleaning Chinese web corpus, providing the NLP community with high-quality Chinese pretraining data and data preparation methods.
• CHC-Bench: A carefully selected multidisciplinary Chinese hard-case benchmark for instruction understanding and following.

Usage

The repository provides a data cleaning pipeline, including:

1. Download the fasttext model to the filter directory, and use the command to filter text. Parameters include input_dir (the directory containing jsonl files), output_dir (the directory for storing filtered jsonl), success_dir (the directory for saving whether file processing succeeded), log_dir (the directory for saving logs), num_worker (the total number of parallel execution processes).
2. Full-text deduplication: provides an example command for running full-text deduplication, where input_dir is the directory storing jsonl files, output_dir is the directory for saving deduplicated jsonl, and content_field_name is the content field name in the jsonl. Partitioned parallel processing can be used to reduce total processing time.
3. Generate Minhash: input_dir is the directory storing jsonl files, output_dir is the directory for saving minhash values, num_worker is the total number of processes, and content_field_name is the content field name.
4. Generate duplicate pairs: input_dir is the directory storing minhash values (the same as the output_dir for generating Minhash), and output_dir is the directory for saving duplicate pair files.
5. Generate connected components: input_dir is the directory containingDirectory of duplicate line files (same as the output_dir for generating duplicate pairs), output_file is the file path for saving connected component information, num_workers is the total number of processes.
6. Generate duplicate line IDs for each file: input_file is the file containing connected component information (same as the output_file for generating connected components), output_dir is the directory for saving duplicate line ID information for each file.
7. Deduplication: input_dir is the directory containing the original jsonl files (same as the input_dir for generating Minhash), output_dir is the directory for saving the deduplicated jsonl, dup_line_id_dir is the directory containing the duplicate line IDs of all jsonl files (same as the output_dir for generating duplicate line IDs for each file).
8. Similar line deduplication: input_dir is the directory containing jsonl files, output_dir is the directory for saving the deduplicated jsonl, num_worker is the total number of processes, content_field_name is the content field name. For detailed pretraining code and configuration, please refer to the Neo repository.

Applicable Scenarios

• Pretraining research for Chinese large language models
• Chinese web corpus cleaning and deduplication
• Evaluation of Chinese instruction comprehension and following capabilities

Limitations and Disclaimers

• This model was developed for academic purposes and uses training data that has undergone strict compliance checks. However, due to the inherent complexity of the data and the broad range of model applications, absolute accuracy or appropriateness of the model's output in all scenarios cannot be guaranteed.
• The model and its related training data are for academic research only. We explicitly disclaim any responsibility for improper use, misinterpretation, illegal activities, dissemination of false information, or any data security issues related to the use of the model or training data.
• Users are encouraged to report any concerns related to data misuse, security vulnerabilities, or potential infringement issues directly so that they can be investigated and resolved promptly.
• Contact: ge.zhang@uwaterloo.ca; duxinrun2000@gmail.com.
• The MAP-CC dataset is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0) license. By using this dataset, you accept and agree to be bound by the terms of this license. This license allows users to share (copy and redistribute the MAP-CC dataset in any medium or format) for non-commercial purposes only, and no modifications or derivatives may be made, with attribution to the creator required. This license was chosen to promote academic and educational uses, spreading knowledge while protecting the creator's work from unauthorized commercial use or modification.
• As of the time the materials were provided, no specific fee or pricing information is mentioned in the text.