MINI_LLM is the author's personal code repository for practicing and reproducing a small-parameter Chinese large model, used for experimenting with and reproducing the pretraining process of an LLM. The project references open-source projects such as Phi2-mini-Chinese, baby-llama2-chinese, and ChatLM-mini-Chinese.
Purpose and Pipeline The repository builds a complete LLM code pipeline including pretraining, SFT instruction fine-tuning, and DPO, and includes DeepSpeed distributed training techniques. The PPO part is marked as to be done. A pretrained model with approximately 1.4B parameters was trained, the base model is QWEN, and the number of training tokens is about 8B.
Model Base The model's base uses the QWEN model, for reasons including: 1. It is a mature open-source Chinese large model project; 2. The author directly used QWEN's tokenizer because its compression rate is relatively good, and therefore also directly used its model.
Pretraining Corpus The following datasets were used for pretraining:
- Wiki Chinese Encyclopedia: wikipedia-cn-20230720-filtered
- BaiduBaiKe: Baidu Netdisk, extraction code: bwvb
- Tiangong Dataset:https://huggingface.co/datasets/Skywork/SkyPile-150B/tree/main/data Data preprocessing adopts QWEN's general approach, adding the end symbol <|im_end|> at the end to distinguish it from the next article. If an article exceeds the specified length, it is truncated, and the truncated part is used as the next sample.
SFT Fine-Tuning The purpose of fine-tuning is to elicit the knowledge in the pretrained model. The project mainly fine-tunes on two types of SFT corpora:
- Daily Q&A SFT data: alpaca-zh, self-instruct data obtained based on GPT4 with reference to the Alpaca method, about 50,000 entries; bell, part of the SFT data from BelleGroup, containing about 3 million Chinese instruction data entries generated by the BELLE project. Since the SFT corpus is generally small, it is not tokenized in advance; instead, during Dataloader construction, batches are tokenized and sent to the model, see sft.py.
DPO DPO stands for Direct Preference Optimization. The code references the 0.2B small Chinese dialogue model ChatLM-Chinese-0.2B. The pipeline includes: Step 1: For instructions that are not quite suitable in the SFT stage, construct an instruction dataset to be optimized (SFT corpora such as alpaca and bell can also be used directly), and provide the data structure. Step 2: Merge the input parts, generate the target dataset chosen, and provide the data structure. Step 3: Through the SFT model from Step 2, input a prompt (such as "Describe the atomic structure.") and obtain the result "An atom is composed of protons, neutrons, and electrons, which are arranged in a specific way into an atomic nucleus.", thereby constructing the rejected dataset, and provide the data structure. Step 4: Merge the input results of Step 2 and Step 3, and provide the data structure. Usage: Step 1: Use the dpo_train file, modify the DpoConfig class in it, and set the corresponding SFT path and training dataset path; Step 2: Execute dpo_train.
Viewing Training Logs You can conveniently view training log changes through SwanLab and use commands to start the local dashboard. If you want to save the training process for viewing in the cloud, you can set the mode variable in swanlab_callback = SwanLabCallback(..., mode="local") to "cloud" at the callback call site.
ModelWeight Download
- Pretrained weights:https://huggingface.co/Lil2J/mini_llm/tree/main
- SFT model weights:https://huggingface.co/Lil2J/mini_llm_sft/tree/main
- DPO model weights:https://huggingface.co/wtxfrancise/mini_llm_dpo/tree/main Also provided are download links for the DPO data and the SFT model to be optimized:https://pan.baidu.com/s/1GYeR6qrUhjsmpgh8-ABDpQ Extraction code: dba9
Training data usage The author first ran the Wiki Chinese Encyclopedia + BaiduBaiKe pretraining corpus, and then ran the first 20 files of the Tiangong dataset as the pretraining corpus; the fine-tuning corpus used alpaca data + bell's train_2M_CN, train_1M_CN, and train_0.5M_CN; the DPO corpus used alpaca data + bell's train_1M_CN.
Applicable scenarios and limitations This project is suitable for individual developers who want to learn or reproduce the pretraining, SFT, and DPO processes of Chinese large models. The limitations are: PPO has not yet been completed; some datasets need to be obtained through Baidu Netdisk; specific training hardware requirements, training duration, complete environment dependency installation commands, and other information are not provided, and are not yet available.
