What is dots.llm1
dots.llm1 is a medium-scale Mixture of Experts (MoE) text large model open-sourced by Xiaohongshu hi lab, with 142 billion parameters and 14 billion activated parameters. The model was pre-trained on 11.2T high-quality token data, using efficient Interleaved 1F1B pipeline parallelism and Grouped GEMM optimization techniques, significantly improving training efficiency. Based on a carefully designed data processing pipeline and two-stage supervised fine-tuning, dots.llm1 performs excellently on Chinese and English general scenarios, mathematics, code, and other tasks, and is highly competitive compared with models such as Qwen2.5-72B. The hi lab team has open-sourced the checkpoint every 1T token in the Pretrain stage and the Instruct model, providing a rich research foundation for the large model community and helping to promote the development of large model technology.
Main features of dots.llm1
- Multilingual text generation: Supports generating high-quality Chinese and English text, suitable for various scenarios such as writing assistance and content creation.
- Complex instruction following: Understands and executes complex instructions to complete specific tasks such as data organization and code generation.
- Knowledge question answering: Provides accurate knowledge question answering services to help users obtain the information they need.
- Mathematical and code reasoning: Has mathematical calculation and code reasoning capabilities, solving math problems and writing simple code.
- Multi-turn dialogue: Supports multi-turn dialogue, engaging in natural and fluent communication with users, understanding context and giving appropriate responses.
Technical principles of dots.llm1
- Mixture of Experts (MoE) architecture: dots.llm1 is an MoE model based on a Decoder-only Transformer, using a 6in128 Expert configuration, meaning each token activates at most 6 experts, with 128 experts in total. Among 142 billion parameters, only 14 billion parameters are activated in each forward pass, and the most relevant experts are selected for computation based on a gating mechanism, improving computational efficiency.
- Efficient data processing and training framework: Pre-trained on 11.2T high-quality token data, with data sources including Common Crawl and web data crawled by its own Spider, and ensuring data quality through multiple rounds of cleaning and filtering. Based on Interleaved 1F1B pipeline parallelism with AlltoAll overlap and an efficient Grouped GEMM implementation, training efficiency is significantly improved.
- Two-stage supervised fine-tuning: Conducts two rounds of basic training on the full dataset, using techniques such as sampling and dynamic learning rate adjustment to initially unlock the model's potential. For specific domains such as mathematics and code, rejection sampling fine-tuning is introduced, combined with a verifier to screen high-confidence important samples, further improving the model's reasoning performance.
- Learning rate scheduling and hyperparameter optimization: Based on the WSD learning rate scheduling method, the learning rate is kept at 3e-4 during the stable phase to train on 10T token corpus, and the annealing phase is divided into two stages to train on 1.2T token corpus, gradually reducing the learning rate. During training, hyperparameters such as batch size are adjusted to ensure training stability and avoid loss spikes.
dots.llm1 project address
- GitHub repository：https://github.com/rednote-hilab/dots.llm1
- HuggingFace model library：https://huggingface.co/rednote-hilab
- arXiv technical paper：https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf
Application scenarios of dots.llm1
- Content creation: Assists in writing copy, stories, news, etc., providing creative inspiration and writing suggestions.
- Education and learning: Used in language learning, programming education, and knowledge question answering to help learners improve their abilities.
- Business Office: Build intelligent customer service, generate data analysis and market research reports, and empower enterprise operations.
- Programming Development: Generate code snippets and code documentation, provide debugging suggestions, and improve development efficiency.
- Personal Assistant: Manage schedules, plan tasks, organize information, and improve personal work and life efficiency.