mindspeed-llm-data-prep

MindSpeed-LLM Data Preprocessing

This Skill guides users in preparing data for MindSpeed-LLM training, covering three data formats: pretraining, instruction fine-tuning, and preference alignment.

Quick Start

Instruction Fine-tuning Data (Alpaca Format)

python preprocess_data.py \
    --input ./dataset/alpaca.parquet \
    --tokenizer-name-or-path ./model_from_hf/Qwen2.5-7B-Instruct/ \
    --output-prefix ./finetune_dataset/alpaca \
    --handler-name AlpacaStyleInstructionHandler \
    --tokenizer-type PretrainedFromHF \
    --workers 4 \
    --prompt-type qwen

Pretraining Data

python preprocess_data.py \
    --input ./dataset/raw_text.parquet \
    --tokenizer-name-or-path ./model_from_hf/Qwen2.5-7B/ \
    --output-prefix ./pretrain_dataset/data \
    --tokenizer-type PretrainedFromHF \
    --handler-name GeneralPretrainHandler \
    --json-keys text \
    --workers 4

Data Handlers

Handler	Input Format	Purpose
GeneralPretrainHandler	{"text": "..."}	Pretraining
AlpacaStyleInstructionHandler	{"instruction": "...", "output": "..."}	Single-turn instruction fine-tuning
SharegptStyleInstructionHandler	{"conversations": [...]}	Multi-turn dialogue fine-tuning
AlpacaStylePairwiseHandler	{"question": "...", "chosen": "...", "rejected": "..."}	DPO preference alignment (Alpaca style)
SharegptStylePairwiseHandler	{"conversations": [...], "chosen": "...", "rejected": "..."}	DPO preference alignment (ShareGPT style)

Input Data Format

Alpaca Format

{
    "instruction": "描述一下量子计算的基本原理",
    "input": "",
    "output": "量子计算利用量子比特..."
}

Supported fields:instruction(required),input(optional),output(required).

ShareGPT Format

{
    "conversations": [
        {"from": "human", "value": "你好"},
        {"from": "gpt", "value": "你好！有什么可以帮助你的吗？"},
        {"from": "human", "value": "请介绍一下深度学习"},
        {"from": "gpt", "value": "深度学习是机器学习的一个分支..."}
    ]
}

Role identifiers:human / gpt(or user / assistant）。

Pairwise Format (DPO)

{
    "system": "You are a helpful assistant.",
    "question": "解释什么是机器学习",
    "chosen": "机器学习是人工智能的一个子领域...",
    "rejected": "机器学习就是让机器学习。"
}

Pretraining Data

{"text": "这是一段用于预训练的长文本..."}

Supports .jsonl、.json、.parquet、.csv、.txt、.arrow input formats.

Key Parameters

Parameter	Description	Example
--input	Input data file path	./dataset/alpaca.parquet
--output-prefix	Output file prefix	./finetune_dataset/alpaca
--tokenizer-name-or-path	Tokenizer path (HF model directory)	./model_from_hf/Qwen2.5-7B-Instruct/
--tokenizer-type	Tokenizer type	PretrainedFromHF
--handler-name	Data handler	AlpacaStyleInstructionHandler
--prompt-type	Dialogue template type	qwen、llama3、glm4
--json-keys	Pretraining text field name	text(required for pretraining data)
--map-keys	Field mapping (non-standard data formats)	'{"prompt":"question"}'
--workers	Number of parallel processing threads	4
--seq-length	Sequence length (used when packing)	4096
--log-interval	Log printing interval	1000

Prompt Templates

Template Name	Applicable Models
qwen	Qwen、Qwen2.5
qwen3	Qwen3
llama3	LLaMA 3/3.1
llama2	LLaMA 2
glm4	GLM-4
deepseek3	DeepSeek-V3
deepseek2	DeepSeek-V2
baichuan2	Baichuan2
mistral	Mistral
chatml	ChatML general format

Output File Structure

Instruction Fine-tuning Data Output

Preprocessing generates packed format files:

<prefix>_packed_input_ids_document.bin / .idx
<prefix>_packed_labels_document.bin / .idx
<prefix>_packed_attention_mask_document.bin / .idx

Pairwise Data Output (DPO)

<prefix>_packed_chosen_input_ids_document.bin / .idx
<prefix>_packed_chosen_labels_document.bin / .idx
<prefix>_packed_rejected_input_ids_document.bin / .idx
<prefix>_packed_rejected_labels_document.bin / .idx

Pretraining Data Output

<prefix>_text_document.bin / .idx

Note: Pretraining --data-path must include the _text_document suffix (e.g., ./dataset/data_text_document), while fine-tuning only needs the prefix.

Data Path Conventions

Important: During training, --data-path should be set to the output prefix (e.g., ./finetune_dataset/alpaca），do notinclude _packed. The data loader get_packed_indexed_dataset() will automatically append the _packed_*_document suffix.

# 正确
--data-path ./finetune_dataset/alpaca

# 错误
--data-path ./finetune_dataset/alpaca_packed

Multi-file Datasets

Supports mixing multiple data files by weight:

--data-path 0.7 ./dataset/pretrain_zh 0.3 ./dataset/pretrain_en

FAQ

Q: Preprocessing error: tokenizer not found

Ensure --tokenizer-name-or-path points to a directory containing tokenizer.json or tokenizer.model .

Q: Processing speed is slow

Increase the --workers parameter (recommended not to exceed the number of CPU cores).

Q: ShareGPT format role identifiers do not match

Ensure to use human / gpt or user / assistant as the role identifier.

Usage order

After data preprocessing is complete:

1. Weight conversion → Use mindspeed-llm-weight-prep
2. Training startup → Use mindspeed-llm-training

Reference resources

• Detailed data format description - Detailed field descriptions and advanced usage for each format
• MindSpeed-LLM repository