InternEvo

Purpose

InternEvo is an open-source lightweight training framework designed to support model pretraining and does not require extensive dependencies. It uses a single codebase to cover two types of scenarios: pretraining on large-scale clusters with thousands of GPUs, and fine-tuning on a single GPU.

Key Capabilities

• A single codebase supports both large-scale cluster pretraining and single-GPU fine-tuning.
• According to official materials, it achieves nearly 90% acceleration efficiency when training on 1024 GPUs.
• Supports a variety of parallelism and training techniques, including: 1F1B Pipeline Parallel, PyTorch FSDP Training, Megatron-LM Tensor Parallel (MTP), Megatron-LM Sequence Parallel (MSP), Flash-Attn Sequence Parallel (FSP), Intern Sequence Parallel (ISP).
• Provides Memory Profiling, conversion of ckpt to HF format, and restoration of ckpt from HF format.
• Provides a raw data Tokenizer and an Alpaca data Tokenizer.
• 2024/08/29: Added support for streaming datasets in huggingface format, and added detailed explanations of the data stream.
• 2024/04/17: Added support for training models on NPU-910B clusters.
• Based on this framework, large language models such as the InternLM-7B series and InternLM-20B series continue to be released.

Usage

The basic official workflow is as follows:

1. Install the specified versions of torch, torchvision, torchaudio, and torch-scatter.
2. Install InternEvo.
3. Optional: If you need to use flash-attention to accelerate training and the environment supports it, install flash-attention (version v2.2.1). For more detailed installation environment or source installation instructions, see the Install Tutorial.
4. Prepare a training script, such as train.py. See the Training Tutorial for details.
5. Prepare training or fine-tuning data: datasets can be downloaded from huggingface, for example roneneldan/TinyStories; download the tokenizer to a local path, for example from https://huggingface.co/internlm/internlm2-7b/tree/main download special_tokens_map.json, tokenizer.model, tokenizer_config.json, tokenization_internlm2.py, and tokenization_internlm2_fast.py to the local path /mnt/petrelfs/hf-internlm2-tokenizer, then modify the configuration file according to the instructions. For the preparation of other types of datasets, see the Usage Tutorial.
6. The configuration file content uses 7B_sft.py as an example; see the Usage Tutorial for details.
7. Start training in a slurm or torch distributed environment. The slurm example is 2 nodes with 16 GPUs; the torch example is 1 node with 8 GPUs.

Applicable Scenarios

• Large language model pretraining on large-scale clusters.
• Model fine-tuning on a single GPU.
• Scenarios that require the use of HuggingFace-format streaming datasets, or that need to train on NPU-910B clusters.
• Developers and researchers who wish to conduct training or research based on the InternLM series models.

Limitations

The materials do not provide information on licensing, specific pricing models, minimum hardware requirements, the upper limit of supported model sizes, or complete parameter descriptions for each feature; missing facts are not provided for now. Installation requires the specified versions of torch, torchvision, torchaudio, and torch-scatter, and more details such as source installation require reference to the Install Tutorial.

Architecture and Acknowledgments

For system architecture details, see the System Architecture document. The project is jointly contributed by Shanghai AI Laboratory and researchers from different universities and companies, with special thanks to the two open-source projects flash-attention and ColossalAI.