1.5-Pints

1.5-Pints is a compact large language model project developed by Pints.AI, aiming to promote reproduction, experimentation, and further development of the model by the open-source community through publicly releasing the model architecture, training scripts, and utility tools. This repository contains the codebases for two models, 1.5-Pints and 0.12-Pint.

Purpose

This project is mainly used for pretraining, fine-tuning, and evaluation of large language models, with particular focus on training compact models using high-quality data within a limited time. According to the materials, 1.5-Pints is a compact LLM pretrained within 9 days.

Main Capabilities

• Provides model architecture and training scripts, supporting users in reproducing the training process.
• Includes data preparation scripts for processing the pretraining dataset (pints-ai Expository Prose pretrain dataset).
• Supports model training, fine-tuning, weight conversion, direct preference optimization (DPO), and evaluation.
• Supports converting the model to HuggingFace format (PyTorch and safetensors), and the use of safetensors files is recommended.
• Provides test code to facilitate users in verifying code stability after modifications.

Usage

Environment requirements: Ubuntu 22.04 LTS x86-64 is recommended, and Debian 12 has also been tested and is usable. arm64/aarch64 is not supported because xformers does not support ARM64 processors. Installing system-level CUDA via apt is not recommended; it is recommended to install the CUDA toolkit in a conda environment.

Step overview:

1. Clone the repository.
2. Create a conda environment using Python 3.10 (3.12 will cause many problems, and 3.11 has not been tested).
3. Install the CUDA toolkit.
4. Install dependencies (note that pip installation of dropout_layer_norm may take about 30 minutes).
5. Install git-lfs and download the pretraining dataset.
6. Prepare the dataset using the scripts in the prepare_dataset folder.
7. Train the model: select the architecture configuration via --model_name (it must be defined in lit_gpt/config.py), adjust --devices and --gpus (they must be the same), and set micro_batch_size to optiOptimize GPU memory, modify TRAIN_DATA_CONFIG to select the training dataset.
8. Fine-tune the model: first convert the pretrained weights, then perform fine-tuning.
9. Run Direct Preference Optimization (DPO).
10. Evaluate the model.
11. Convert the lit model to a HuggingFace model (PyTorch and safetensors).
12. Perform inference using HuggingFace.

Use Cases

Suitable for researchers and developers conducting training and fine-tuning experiments on large language models, as well as for the open-source community to reproduce and improve compact LLMs.

Limitations

• ARM64 architecture is not supported.
• Specific environment configuration is required (Ubuntu 22.04 or Debian 12, Python 3.10, conda environment).
• Training and fine-tuning require certain hardware resources (GPU).
• The materials do not provide specific model performance metrics, pricing information, or prices.
• The materials do not provide detailed model use cases or application scenarios.