MicroLlama is a small large language model pretraining project initiated by an individual developer, with the goal of training a 300M parameter Llama model from scratch under limited computational resources and budget. The project follows strict constraints: a total budget of $500, must use fully open-source datasets and models, and is not allowed to fine-tune existing models or use GPT-4 etc. to generate training data.
Purpose
MicroLlama is mainly used to explore the feasibility of building usable language models under limited computing power conditions. It can serve as the foundation for lightweight models, used for tasks such as text embedding, sentence similarity calculation, lightweight chatbots, etc., and is also suitable as a reference for research on pretraining and evaluation of small language models.
Main capabilities
- Modified based on the TinyLlama project, supports pretraining a 300M parameter model.
- Uses the Slimpajama dataset, with Starcoderdata removed, so the model itself does not have code capabilities unless subsequently fine-tuned.
- Added the ability to process and tokenize Slimpajama when downloading data, saving time and storage.
- Provides auxiliary scripts and Python code, such as uploading pretrained checkpoints to Hugging Face Hub.
- The project has derived MicroLlama-text-embedding (text embedding model) and MicroLaVa (lightweight visual question answering model, based on TinyLLaVA_Factory, with a total of 700M parameters, which can be pretrained and fine-tuned on a single Nvidia RTX 4090).
- Has been officially supported by LitGPT.
Usage
The project source code, installation dependencies, and evaluation commands can all be found in the GitHub repository. Users can refer to the instructions in the repository to install dependencies and use lm-evaluation-harness to reproduce the evaluation results. Model checkpoints can be uploaded to Hugging Face Hub.
Applicable scenarios
- As the foundation for lightweight models, used for tasks such as sentence transformer, BERTScore, etc., replacing BERT series models.
- After fine-tuning, it can be used to build lightweight chatbots.
- Suitable for individual researchers or teams with limited resources to conduct small language model pretraining experiments.
Limitations
- The model scale is relatively small (300M), and its performance on standard benchmarks is inferior to TinyLlama-1.1B, butit is superior to google-best/bert-large-uncased (336M), except for the ARC_c dataset.
- Due to the removal of Starcoderdata, the model cannot directly perform code-related tasks.
- The project is still in progress, and some features (such as the forked code after fine-tuning) have not yet been cleaned up and released.
- As of the time the information was provided, $280 has been spent on computing (4 Nvidia 4090s trained on Vast.ai for 4 days, 50B tokens), and $3 on AWS S3 storage.
- The model has not been fine-tuned, and direct use for downstream tasks requires additional training.
Evaluation results
Evaluated using the standard settings of lm-evaluation-harness, all datasets use the acc_norm metric, and winogrande and boolq use acc. Compared with google-best/bert-large-uncased, MicroLlama performs better on most datasets, but is slightly inferior on ARC_c. The evaluation results can be used to verify the project settings; please refer to the repository for specific values.
