GPT-NeoX is a library developed by EleutherAI for training large-scale language models on GPUs. Its current framework is based on NVIDIA's Megatron language model and incorporates DeepSpeed technologies as well as some novel optimizations. The goal of this project is to provide a centralized and easily accessible collection of techniques for training large-scale autoregressive language models, accelerating research on large-scale training.
Key capabilities:
- Supports distributed training with ZeRO and 3D parallelism.
- Supports multiple systems and hardware, can be launched via Slurm, MPI, and IBM Job Step Manager, and has been run at scale on platforms such as AWS, CoreWeave, ORNL Summit and Frontier, LUMI, and CINECA Leonardo.
- Includes cutting-edge architectural innovations such as rotary and alibi positional embeddings, parallel feedforward attention layers, and flash attention.
- Provides predefined configurations for popular architectures, including Pythia, PaLM, Falcon, and LLaMA 1 and 2.
- Supports Curriculum Learning.
- Easy to connect with the open-source ecosystem, including Hugging Face's tokenizers and transformers libraries, monitor experiments via WandB / Comet / TensorBoard, and evaluate with the Language Model Evaluation Harness.
Version information: Before March 9, 2023, GPT-NeoX depended on DeeperSpeed based on the old version of DeepSpeed (0.3.15). In order to migrate to the latest upstream DeepSpeed version while allowing users to access the old versions of GPT-NeoX and DeeperSpeed, the project introduced two versioned releases:
- Version 2.0 of GPT-NeoX and DeeperSpeed is the latest version built on the latest DeepSpeed and will continue to be maintained.
- Version 1.0 of GPT-NeoX and DeeperSpeed preserves snapshots of the old stable versions used when training GPT-NeoX-20B and the Pythia Suite.
Usage: The codebase is mainly targeted at Python 3.8-3.10 and PyTorch 1.8-2.0 for development and testing. It depends on DeeperSpeed (the project's fork of the DeepSpeed library with additional modifications), and it is recommended to use Anaconda or a virtual machine for environment isolation. Supports AMD GPUs (MI100, MI250X), with JIT fused kernel compilation; fused kernels are built and loaded on demand. To use Flash-Attention, you need to install the additional dependencies in ./requirements/requirements-flashattention.txt, or run in a PyTorch NGC container with the library preinstalled, and then set the attention type in the configuration. To use Transformer Engine (TE), you need to install the additional dependencies in ./requirements/requirements-transformer-engine.txt. Supports multi-node training, and you can use launchers such as pdsh (default), MPI (supporting mvapich, openmpi, mpich, impi), or Slurm. You need to configure a hostfile containing IP addresses and the number of GPU slots, or specify the path in the environment variable DLTS_HOSTFILE. Provides Apptainer (formerly Singularity) and Docker containers, located in the containers directory. All functionality is launched through deepy.py (a wrapper around the deepseed launcher), providing three main entry points: train.py for training and fine-tuning; eval.py for evaluating trained models with the language model evaluation harness; generate.py for sampling text from trained models. For example, to start training: python3 deepy.py train.py /path/to/configs/my_model.yml
Parameters are passed to deepy.py through YAML configuration files, and the configs directory contains various example .yml files.
Use cases: This library has been used by more than 40 organizations (academia, industry, government) to train more than 50 models, such as EleutherAI's Pythia, Stability AI's StableLM, Together.ai RedPajama-INCITE, Oak Ridge National Laboratory's FORGE, SB Intuitions' Sarashina1, Tilde's TildeOpen-30B. Arc Institute's Evo 2 was trained using the Savanna framework derived from GPT-NeoX. Suitable for scenarios that require training models with billions of parameters from scratch.
Limitations: If the user does not intend to train models with billions of parameters from scratch, this library may not be suitable. For general inference needs, the official recommendation is to use the Hugging Face transformers library, which supports GPT-NeoX models.
Other features: Supports Dropless Mixture of Experts (DMoE) through the megablocks library, compatible with existing Megatron tensor parallelism and DeepSpeed pipeline parallelism setups. This implementation leverages existing tensor parallelism groups to shard expert weights, and uses Sinkhorn routing to avoid load balancing issues.
