Baichuan-M1-14B is the industry's first open-source large language model developed by Baichuan Intelligence from scratch specifically optimized for medical scenarios. While maintaining excellent general capabilities, it has strong performance in the medical field. According to official information, the model achieves the performance of models of the same size on most general benchmark evaluations, while in medical scenarios it reaches the level of models 5 times or even larger.
[Main Capabilities] The model is trained on 20 trillion tokens of high-quality medical and general data, conducts fine-grained medical professional modeling for more than 20 medical departments, introduces innovative model structures, significantly improves context understanding and long-sequence task performance, and provides two versions: the Base model and the Instruct model.
[Medical Data Construction] Detailed data collection and synthesis were carried out for the medical field, including: tens of millions of professional medical data (Chinese and English professional papers, medical cases, medical textbooks, knowledge bases, etc.); hundreds of millions of medical question-answering and clinical data (covering complex medical reasoning and actual clinical cases); comprehensive data classification and evaluation (classified by medical department, content, and value to ensure balanced data distribution); synthetic data design (using medical knowledge texts such as textbooks, guidelines, knowledge bases, and papers, as well as structured knowledge graphs and clinical medical records, to generate diverse high-quality medical reasoning data); self-reflection mechanisms and reward models (continuously improving the quality of synthetic data, ultimately generating nearly one trillion tokens of medical data, covering long-tail knowledge and complex scenarios).
[General Data Construction] A 20T multilingual general dataset, including 14T English data, 4T Chinese data, and 2T data covering 30 mainstream languages; deduplication and upsampling strategies are adopted; based on 27 categories of global knowledge classification, the data mixture is optimized through small-model experiments to ensure a balance between general capabilities and domain capabilities.
[Model Structure Innovation] By introducing lightweight short convolution operations when computing Key and Value, it significantly reduces the standard Transformer model's dependence on learning the induction heads mechanism and improves in-context learning capability; a sliding window attention mechanism is used in some layers to reduce KV Cache memory usage, achieving a balance between computational efficiency and performance, and is suitable for long-sequence tasks; by increasing the dimensions of some attention heads, RoPE curve oscillation is reduced, making long-sequence task performance more stable; a WSD learning rate scheduling strategy is adopted, using a high peak learning rate to promote model generalization ability; dynamic gradient clipping dynamically chooses whether to skip updates when gradients are too large, reducing instability.
[Training and Alignment] A multi-stage curriculum learning and alignment optimization method is adopted. Training is divided into three stages: General capability improvement stage, basic medical knowledge improvement stage, and advanced medical knowledge improvement stage. Alignment optimization improves generation quality, logical reasoning ability, and user preference alignment through reinforcement learning and partial-order pair data, including ELO (optimizing diverse high-quality chain-of-thought generation based on maximum likelihood), TDPO (using partial-order pair data to optimize the generation model), and PPO (enhancing generation logic and task performance through policy optimization).
[Usage] It is recommended to use the latest version of the Hugging Face Transformers library (at least 4.47.0). Official code snippets for using the Baichuan-M1-14B-Instruct model are provided. The documentation mentions that inference support for Baichuan-M1 is being submitted to the vLLM community, and the official vLLM version can be used later. The MindIE framework supports inference of Baichuan-M1 on Huawei Ascend chips; for usage, refer to MindIE Baichuan-M1-14B-Base. The openMind fine-tuning suite supports fine-tuning of this series of models; for usage, refer to openMind Baichuan-M1-14B.
[Commercial Use and Compliance] If the Baichuan-M1 model or its derivatives are used for commercial purposes, the licensor must be contacted: opensource.contact@baichuan-inc.com. For specific information, see the "Baichuan-M1-14B Model Community License Agreement." All users must comply with laws and regulations and must not use the model for purposes that endanger national security or violate the law. The official citation format is also provided.
[Applicable Scenarios and Limitations] This model is suitable for reasoning, question answering, and clinically related tasks in medical scenarios, and also has general language capabilities that can be used for general language modeling. Regarding its limitations, the official materials do not list specific shortcomings in detail; support for some frameworks (such as vLLM) is still being submitted, and specific availability has not yet been provided.