SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsbuild_MiniLLM_from_scratch

build_MiniLLM_from_scratch

build_MiniLLM_from_scratch is an open-source project whose goal is to build a small-parameter LLM from scratch, completing all four stages—pretraining, instruction fine-tuning, reward model, and reinforcement learning—to achieve a chat model capable of simple chat tasks at a controllable cost. The project uses the bert4torch training framework, with concise and efficient code. The trained checkpoints can seamlessly connect with the transformers package for inference, and the file reading method during training has been optimized to reduce memory usage. Currently, the pretraining and instruction fine-tuning stages have been completed, and DPO alignment is still being tested. The project provides complete training logs for reproduction and comparison, and also includes a self-cognition dataset, allowing customization of attributes such as the robot's name and author. The chat model supports multi-turn conversations. It should be noted that, limited by corpus size, model scale, and SFT corpus quality, the model currently only has simple chat capabilities and does not have the ability to answer complex questions.

Open website

About this tool

Project Positioning

build_MiniLLM_from_scratch is an open-source project that builds MiniLLM from 0 to 1. The practice route covers four stages: pretraining (pretrain) -> instruction fine-tuning (sft) -> reward model -> reinforcement learning (dpo in practice). The goal is to complete, at a controllable cost, a chat model capable of simple chat tasks. The first two stages have currently been completed.

Main Capabilities and Features

  • Uses the bert4torch training framework, with concise and efficient code.
  • The trained checkpoint can seamlessly connect with transformers, and the transformers package can be used directly for inference.
  • Optimized the file reading method during training to optimize memory usage.
  • Provides complete training logs for reproduction and comparison.
  • Adds a self-cognition dataset, allowing customization of attributes such as the robot's name and author.
  • The chat model supports multi-turn conversations.

Updates and Existing Models

  • 20240403: Added MiniLLM-0.2B-SFT trained on 11.57 million samples, supporting multi-turn conversations.
  • 20240325: Added the 1.1B model (from zRzRzRzRzRzRzR).
  • 20240316: Initial commit, including the pretrained models MiniLLM-0.2B-NoWudao-Base and MiniLLM-0.2B-Base; the SFT model MiniLLM-0.2B-SFT-Alpaca.

Corpus and Data

The project has open-sourced the pretraining corpus processed with the ChatGLM2-6B tokenizer, totaling 63.4 billion Tokens, with the link being Corpus. It also provides an instruction fine-tuning corpus (filtered usable datasets). The alignment model (DPO) section is also organizing the corresponding corpus (filtered usable datasets), but it is still in the testing stage.

Usage and Limitations

The project does not provide specific installation commands, training steps, or dependency versions in the given materials, and no related content is currently provided. You can directly view the folder and file structure and repository file navigation in the GitHub repository, and combine it with transformers for inference connection.

Statement and Applicable Scenarios

Limited by corpus size, model scale, and the size and quality of the SFT corpus, the model trained in this experiment currently only has simple chat functionality and does not have the ability to answer complex questions. Therefore, it is more suitable as a practical reference for learning the LLM training process from scratch and reproducing pretraining and instruction fine-tuning under low-compute conditions.The project implementation refers to baby-llama2-chinese, and provides WeChat group communication (the group has more than 200 people, so there is an invitation limit; you can add the personal WeChat to be pulled into the group).

Related tools

Browse category