What is Baichuan-M1-14B
Baichuan-M1-14B is the industry's first open-source medical-enhanced large model launched by Baichuan Intelligence. Its medical capabilities surpass those of the larger Qwen2.5-72B and are nearly on par with o1-mini. It is specifically optimized for medical scenarios while also possessing strong general capabilities. The model is trained on 20 trillion tokens of high-quality medical and general data, covering fine-grained professional knowledge across more than 20 medical departments. It performs excellently in medical reasoning and knowledge Q&A, and in medical scenarios its performance can reach the level of models five times its own parameter size. The core advantages of Baichuan-M1-14B lie in its innovative model architecture and training methods. It introduces techniques such as short convolution attention mechanism, sliding window attention mechanism, and optimized positional encoding oscillation, improving context understanding and performance on long-sequence tasks. The model adopts multi-stage curriculum learning and alignment optimization methods, optimizing generation quality and logical reasoning ability through reinforcement learning.
Main Features of Baichuan-M1-14B
- Powerful Medical Reasoning Ability: Baichuan-M1-14B performs excellently in the medical field. Its medical reasoning ability surpasses the larger Qwen2.5-72B-Instruct and is nearly on par with o1-mini. It can handle complex medical problems and provide precise medical reasoning and recommendations.
- Multilingual Support: The model supports both Chinese and English, and can handle multilingual medical data.
- Open Source and Commercially Usable: Baichuan-M1-14B is an open-source model that supports low-cost deployment and multilingual applications. The open-source strategy aims to lower development barriers and promote the construction of the medical AI ecosystem.
- Medical Evidence-Based Mode: The model unlocks a "medical evidence-based mode," which can analyze and integrate evidence of different authority levels through a multi-level evidence grading system to provide reliable medical reasoning. It is based on a self-built evidence-based medicine knowledge base covering massive medical papers, authoritative guidelines, and expert consensus.
- Multi-Domain Reasoning Ability: Baichuan-M1-14B demonstrates comprehensive reasoning ability across multiple domains such as language reasoning, visual reasoning, and search reasoning.
Technical Principles of Baichuan-M1-14B
- Data Collection and Processing
- Massive Medical Data: Model training is based on 20 trillion tokens of high-quality medical and general data, covering 20+ medical departments.
- Data Classification and Evaluation: Data is classified by medical department, content, and value to ensure balanced data distribution.
- Synthetic Data: Generate diverse high-quality medical reasoning data through textbooks, guidelines, knowledge graphs, and clinical medical records.
- Innovative Model Architecture
- Short Convolution Attention Mechanism: By introducing short convolution operations, it reduces dependence on induction heads and improves in-context learning ability.
- Sliding Window Attention Mechanism: Reduces KV Cache memory usage and improves computational efficiency for long-sequence tasks.
- Optimized Positional Encoding Oscillation: Reduces RoPE curve oscillation by increasing the dimensions of some attention heads.
- Multi-Stage Training Method
- General Knowledge Ability Improvement Stage: Improves basic language ability and common sense.
- Medical Basic Knowledge Improvement Stage: Introduces high-quality medical data, focusing on improving reasoning, mathematics, and medical knowledge abilities.
- Advanced Medical Knowledge Improvement Stage: Further optimize data quality, focusing on complex medical reasoning and long-tail knowledge.
- Reinforcement Learning Optimization
- ELO（Exploratory Log-likelihood Optimization）: Optimize the chain-of-thought path to improve generation quality and logical reasoning ability.
- TDPO（Token-level Direct Preference Optimization）: Use preference pair data to optimize the generative model, making it better aligned with user preferences.
- PPO（Proximal Policy Optimization）: Further enhance generation logic and task performance through policy optimization.
- Model Optimization Strategy
- Large Peak Learning Rate Strategy: Adopt the WSD learning rate scheduling strategy to promote model generalization ability.
- Dynamic Gradient Clipping: Reduce instability caused by special samples or steep loss landscapes.
Baichuan-M1-14B Project Address
- GitHub Repository：https://github.com/baichuan-inc/Baichuan-M1-14B
- HuggingFace (Base Model)：https://huggingface.co/baichuan-inc/Baichuan-M1-14B-Base
- Hugging Face (Instruct Model)：https://huggingface.co/baichuan-inc/Baichuan-M1-14B-Instruct
Application Scenarios of Baichuan-M1-14B
- Clinical Decision Support: Baichuan-M1-14B can quickly and accurately answer clinical medical questions through the "medical evidence-based mode." It provides doctors with reliable medical reasoning support and helps improve diagnosis and treatment efficiency.
- Medical Research Support: The model can help researchers quickly obtain authoritative medical evidence and clinical guidelines, shortening research exploration time.
- Patient Health Management: Baichuan-M1-14B can provide patients with personalized health management suggestions, helping them better understand their own health conditions and scientifically manage their lifestyles.
- Scientific Research and Data Analysis: The model's multi-domain reasoning ability can handle complex scientific research problems and provide efficient data analysis support.