What is Phi-4
Phi-4 is a 14B-parameter small language model released by Microsoft, performing excellently in complex reasoning in mathematics and other fields as well as traditional language processing. Phi-4 uses data quality as the core training focus, extensively incorporating synthetic data to improve the model's performance on STEM question answering and math competition problems. Phi-4 introduces a new training paradigm, midtraining, to enhance long-text processing capability, with a window length of up to 16K. Phi-4 performs excellently on programming tasks, leading other open-source models (such as 70BLlama 3.3and 72BQwen 2.5) with 82.6% accuracy on the HumanEval benchmark. Phi-4 is open-sourced on Hugging Face and available for download and use.
Phi-4's main features
- Question answering capability: It can understand and answer various questions, especially performing excellently in the STEM (science, technology, engineering, and mathematics) fields.
- Mathematical problem solving: The model scores over 90 on the American Mathematics Competitions AMC 10/12, demonstrating strong mathematical reasoning ability.
- Programming tasks: Phi-4 performs excellently on programming tasks, can understand and generate code, and solve programming problems.
- Long-text processing: Based on the midtraining stage, Phi-4 can process contexts up to 16K long while maintaining a high recall rate.
- Complex reasoning: The model demonstrates the ability to handle complex reasoning tasks in multiple benchmarks, such as MMLU and GPQA.
- Safe interaction: Phi-4 undergoes safety alignment during post-training to ensure that interactions with users comply with responsible AI principles.
Phi-4's technical principles
- Synthetic data training: Phi-4's training process extensively uses synthetic data, generated based on techniques such as multi-agent prompting, self-revision, and instruction reversal, to improve the model's reasoning and problem-solving capabilities.
- Midtraining stage: A midtraining stage added between pre-training and post-training improves the model's ability to process long texts.
- Long-context data selection: Samples longer than 8K tokens are selected from high-quality non-synthetic text, with samples over 16K tokens weighted to match the target length.
- Contrastive learning: The pivot token search (PTS) method is used to identify the key tokens that have the greatest impact on model output, generating contrastive learning data with a high signal-to-noise ratio.
- Human feedback: Combined with Human Feedback DPO, high-quality positive and negative sample pairs are constructed to make model outputs better align with human preferences.
Phi-4's project addresses
- Project official website：introducing-phi-4
- HuggingFace model library：https://hf-mirror.com/microsoft/phi-4
- arXiv technical paper：https://arxiv.org/pdf/2412.08905
Phi-4's application scenarios
- Educational assistance: As an educational assistance tool, it helps students solve complex problems in the STEM fields and provides tutoring for mathematics and programming homework.
- Technical research: In scientific research, it understands and generates concepts and data in research papers, assisting researchers with literature reviews and data analysis.
- Software development: It assists software development, including code generation, debugging, and feature implementation.
- Intelligent assistant: As an intelligent assistant, it understands and responds to user queries, providing information retrieval, schedule management, and other personal assistant services.
- Enterprise decision support: Analyze large amounts of data to provide enterprises with market trend analysis, risk assessment, and decision support.