DeepSeek-671B-SFT-Guide

Project Overview

DeepSeek-671B-SFT-Guide is an open-source solution for full-parameter fine-tuning of the DeepSeek-V3/R1 671B model. This project was jointly launched by the Institute of Automation, Chinese Academy of Sciences and Beijing Wenge Technology Co., Ltd. It includes complete code and scripts from training to inference, as well as experience and conclusions accumulated in practice.

Main Capabilities

• Implemented modeling files containing the DeepSeek-V3/R1 training logic (located in the ./model directory; the code logic is based on the DeepSeek-V3 paper and the DeepSeek-V2 modeling files).
• Implemented full-parameter fine-tuning of DeepSeek-V3/R1 671B based on data parallelism (DeepSpeed ZeRO) and sequence parallelism (SP).
• Summarized the entire process of model training and deployment, including the pitfalls encountered, problems, and solutions.

Usage

Hardware Configuration

See the project table for the configuration of a single server. The cluster has 32 machines with the same configuration, sharing 100TB of storage space mounted at /nfs. The operating system is Ubuntu 22.04, machines communicate with each other over an IB network, GPUs communicate over NVLink, and the CUDA version is 12.6.

Environment Setup

The project extends and improves the xtuner framework to support full-parameter fine-tuning of DeepSeek V3/R1 (i.e., the DeepseekV3ForCausalLM model architecture), supporting data parallelism (DP based on DeepSpeed ZeRO) and sequence parallelism (SP). A Python environment needs to be installed, dependencies need to be installed according to the requirements.txt in the project, and the core code related to DeepseekV3ForCausalLM training in ./code/xtuner needs to be copied over the corresponding code in the original xtuner package.

Data Preparation

The project extends the OpenAI standard data format to be compatible with reasoning data. The format of each piece of raw training data is as follows: if there is a reasoning process, the reasoning_content field of the assistant role is not empty. To simplify the processing logic, reasoning_content and content are merged into the content field according to the DeepSeek training format. In addition, to be compatible with the multi-turn dialogue training logic, for each turn asThe sistant role adds a loss field, and loss is calculated only for content whose loss value is true. The project provides an example file of the converted data, ./data/train_example.json, for reference. During actual training, the program will automatically convert it to a specific format according to the DeepSeek V3/R1 training template (shown here only for illustration).

Start Training

The project provides training code and startup scripts:

• ./code/scripts/sft_deepseek.py: the SFT training configuration file, including hyperparameter settings, model and tokenizer configuration, training strategy, etc. Modify the model training-related configuration in this file.
• ./code/scripts/sft_deepseek.sh: the SFT training startup script, which is a single-node execution file and needs to be executed on each machine via slurm or pdsh. For each machine, the only difference in the training startup command is the NODE_RANK value. If there are 32 machines, the range is from 0 to 31.

Taking pdsh as an example, the steps to start training are as follows:

1. Overwrite the corresponding original file downloaded from platforms such as huggingface with the modeling_deepseek.py file provided in the project's ./model directory;
2. Use pdsh to start training. On machine 0, execute the command pdsh -R ssh -w node[0-31] 'bash ./code/scripts/sft_deepseek.sh' to start the full-parameter fine-tuning task of the model on 32 machines. Modify node[0-31] according to the machine hostname or IP address. During training, the training process (loss changes, etc.) can be visualized through tensorboard.

The project provides several experimental conclusions, including the feasibility of model training under different parallel strategies and configurations. The training data is about 100k, and the training context length is 32k. The table reports the number of machines used (nodes), sequence parallelism (sp), data parallelism method (dp), per-GPU batch size (bs), number of iterations (epoch), learning rate (lr), per-GPU memory (mem), experimental records, and remarks. The training screenshots show that when fully fine-tuning DeepSeek V3 on the constructed reasoning data, the initial loss is usually around 3.5, and after training for 1 epoch the loss converges to 1.2 or so.

Model Weight Conversion

During training, it is recommended to use at least 100TB of large-capacity SSD storage, because a single pth intermediate result occupies about 7.4TB of disk space. After training, the pth needs to be converted to the huggingface format, which is better compatible with mainstream inference frameworks (such as vllm). On a single machine node, execute bash ./code/scripts/convert_pth_to_hf.sh to complete the model weight format conversion. The pth path and weight save path in the script can be modified according to the actual situation. Note that this process requires a large amount of CPU memory, and virtual memory expansion can be used to prevent insufficient memory. Swap is Linux's virtual memory, used to store part of the data to disk when physical memory (RAM) is insufficient, freeing up RAM.

Model Inference Deployment

According to the DeepSeek V3 Github introduction, there are multiple ways to deploy the model locally. The project used vLLM to conduct a simple deployment test on the fully fine-tuned model. Assume that an environment named vllm has been created according to the official vLLM documentation. If using a slurm cluster, refer to the provided script and execute the sbatch command sbatch ./code/scripts/vllm_deploy_slurm.sh to submit the job. For half-precision (bf16/fp16) models, it is recommended to deploy using 4 machines with 32 cards. If you need to configure the port numbers of ray or the api server, you can modify the sh file yourself. If you need to start deployment via pdsh (assuming four machines node0~node3 are used), refer to the following steps:

1. Set environment variables (node0~node3).
2. Start Ray Head (node0).
3. Start Ray Worker (node1~node3).
4. Start vLLM (node0). After startup, you can use the curl command to test whether the interface starts normally. After a period of time, if the terminal outputs the expected response, it means that the entire process from training to deployment has been successfully completed.

Applicable Scenarios

Suitable for researchers, engineers, and teams who need to perform full-parameter fine-tuning of the DeepSeek-V3/R1 671B model, especially in scenarios of training and deployment in large-scale cluster environments.

Limitations

The project does not clearly explain the specific effects of model fine-tuning,charging situation, price, platform compatibility, and other details. This information has not yet been provided. Users need to evaluate feasibility based on their own hardware conditions and needs.

Acknowledgements

The project acknowledges open-source projects such as DeepSeek-V2/V3/R1, Huggingface transformers, DeepSpeed, Xtuner, vLLM, etc. This project uses the Apache-2.0 license.