What is MiniMind
MiniMind is an open-source ultra-small language model project that helps individual developers train their own language model from scratch at extremely low cost. Based on a lightweight design, the smallest version requires only 25.8M parameters, with a size only 1/7000 of GPT-3, making it suitable for fast training on ordinary personal GPUs. MiniMind provides complete training pipeline code, including pretraining, supervised fine-tuning, LoRA fine-tuning, reinforcement learning, and model distillation, supports multimodal capabilities (such as the vision-language model MiniMind-V), and is compatible with mainstream frameworks such astransformersandpeft. MiniMind has open-sourced high-quality datasets and a custom tokenizer, making it suitable for LLM beginners to get started quickly.
Main features of MiniMind
- Extremely low-barrier model training：
- Low cost: Only 3 RMB in GPU rental cost (based on NVIDIA 3090).
- Fast training: Training from scratch takes only 2 hours.
- Lightweight design: The smallest model has only 25.8M parameters, suitable for running on ordinary devices.
- Full-process open source: Provides complete training code, covering pretraining, supervised fine-tuning (SFT), LoRA fine-tuning, direct preference optimization (DPO), and model distillation. All core algorithms are implemented natively based on PyTorch, without relying on third-party wrappers, making them easy to learn and extend.
- Supports multiple training techniques：
- Mixture of Experts (MoE) architecture: Dynamically allocates computational resources to improve the learning efficiency of small models.
- Direct Preference Optimization (DPO): Optimizes model outputs according to human preferences without requiring a complex reward model.
- Multimodal extension: Supports visual multimodality (MiniMind-V), enabling image dialogue and image-text generation.
Technical principles of MiniMind
- Transformer architecture: Based on a Transformer Decoder-Only structure, similar to the GPT series. It uses pre-normalization (Pre-Norm) and RMSNorm normalization methods to improve model performance. It uses the SwiGLU activation function instead of ReLU to improve training efficiency.
- Mixture of Experts (MoE) technology: Introduces a mixture of experts module in the feed-forward network (FFN), dynamically allocating computational resources to different "experts". Based on sharing and isolation techniques, it improves the learning ability and efficiency of small models.
- Lightweight training pipeline：
- Pretraining (Pretrain): Uses cleaned high-quality text data for unsupervised learning to accumulate language knowledge.
- Supervised fine-tuning (SFT): Performs supervised fine-tuning of the model based on dialogue templates to adapt it to chat scenarios.
- LoRA fine-tuning: Updates a small number of parameters based on low-rank decomposition to quickly adapt to a specific domain or task.
- Direct Preference Optimization (DPO): Optimizes the quality of model outputs based on labeled data of human preferences.
- Model distillation: Mimics the outputs of large models to further improve the performance of small models.
MiniMind project address
- Project official website：https://jingyaogong.github.io/minimind/
- GitHub repository：https://github.com/jingyaogong/minimind
- HuggingFace model library：https://huggingface.co/collections/jingyaogong/minimind
Application scenarios of MiniMind
- AI beginners and students: Students and beginners interested in AI, completing graduation projects or participating in academic research.
- Individual developers and independent developers: Individual developers with limited resources, rapidly developing and deploying AI applications, suitable for experimental and innovative projects.
- Vertical domain professionals: Professionals in fields such as healthcare, law, and education, developing medical consultation assistants, legal consulting tools, or educational tutoring systems.
- Small teams and entrepreneurs: Small teams or entrepreneurs with limited resources, developing a minimum viable product (MVP) or exploring new business directions.
- Hobbyists and creative individuals: Hobbyists interested in technology but without a deep background, developing intelligent chatbots, generating creative content, or exploring multimodal applications.