Zero-Chatgpt is an open-source project aimed at running through the complete ChatGPT technical pipeline from scratch. The project covers the full process from data collection to reinforcement learning, with the main purpose of getting the code and pipeline to run through, while performance tuning can be done later.
Purpose
This project is used to learn and reproduce the complete training pipeline of large language models (LLMs), including pretraining, instruction fine-tuning (SFT), and reinforcement learning from human feedback (RLHF, PPO). It is suitable for researchers and developers who want to deeply understand the ChatGPT technical pipeline.
Main Capabilities
The project provides complete data processing and training code, specifically including:
- Data collection: Collected about 10B Chinese training corpus, with sources including Chinese Wikipedia, Chinese Baidu Baike, and a randomly sampled portion of SkyPile-150B.
- Data cleaning and deduplication: Chinese Baidu Baike was cleaned and deduplicated, removing personal introductions, product introductions, and overly short data, and performing strict deduplication, ultimately reducing from 5.63 million data entries to more than 1.4 million. The code is located in the data_process folder.
- Vocabulary training: The vocabulary size is set to 32000 (referencing llama), and special_tokens are set referencing qwen. The code is located in the train_tokenizer folder.
- Language model pretraining: The model structure references llama, and the code references huggingface. The model size is designed to be 0.1B. The pretraining data generation code, training scripts, and training code are located in the pretrain folder.
- Instruction fine-tuning: Uses firefly-train-1.1M and ruozhiout_qa_cn.jsonl data. After cleaning and deduplication, more than 400,000 entries remain, and 300,000 entries are actually used for training. The scripts and code are located in the sft folder.
- Reinforcement learning (RLHF, PPO): RLHF data was generated based on the data not used by SFT, totaling 100,000 entries. Of these, 50,000 entries are used to train the Reward Model (step 2), and after 1 epoch the classification accuracy can reach 92%; all 100,000 entries are used to train RLHF (step 3). The code references DeepSpeed-Chat with modifications and is located in the rlhf folder.
Usage
The project environment involves cuda 12.1, pytorch, transformers, deepspeed etc. requirements.txt lists the runtime environment dependencies. The compute resources are 2 A40 GPUs, and pretraining takes about 2 days. The training data, model weights, and training image files have been provided. After removing the prefix from the model weight names (changing them to model.safetensors or pytorch_model.bin) and placing them together with the model code and configuration files (under the model folder), they can be loaded. The pretraining data and training image will be uploaded later due to their large size.
Applicable Scenarios
Suitable for learning the full training pipeline of large language models, conducting small-scale model experiments, and research scenarios that scale up by adjusting configurations on this basis. Project experience shows that larger models and more data significantly improve performance.
Limitations
The project's current model size is 0.1B, so actual performance is limited. After using 300,000 data samples in the SFT stage, the performance is still not ideal, and more data may be needed. The RLHF stage performs poorly; as the number of training steps increases, the model's capabilities instead decline, the frequency of refusing to answer increases, but the frequency of repetitive parroting decreases. Lowering the learning rate alleviates this somewhat, but the performance is still poor, possibly due to poor data quality. The project's main goal is to get the pipeline working end to end, and performance tuning remains to be done later.
The project also introduces another open-source image-text multimodal project, Zero-Qwen-VL, which trains an image-text large model with better Chinese support from scratch, using the image encoder of qwen-vl and the language model of Qwen2-0.5B-Instruct, and can be replaced with a larger model when compute resources are sufficient.
