What is Phi-3
Phi-3 is a new generation series of advanced small language models launched by Microsoft Research, including three versions of different scales: phi-3-mini, phi-3-small, and phi-3-medium. While maintaining a relatively small parameter scale, these models achieve language understanding and reasoning capabilities comparable to large models through carefully designed training datasets and optimized algorithms. In particular, the phi-3-mini model, with only 380 million parameters, can outperform models with larger parameter counts in multiple benchmark tests, and due to its compact size, it can even run on smartphones. The development of the Phi-3 series models demonstrates the latest progress in data engineering and model optimization, providing new possibilities for the future development of miniaturized, high-performance AI models.
Technical report:https://arxiv.org/abs/2404.14219
Hugging Face Phi-3 model address:https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3
Ollama Phi-3 model address:https://ollama.com/library/phi3
Phi-3's model series
- phi-3-mini: The smallest language model, with 380 million parameters. Despite its compact size, its benchmark performance on multiple language understanding tasks is comparable to models with larger parameter counts such as Mixtral 8x7B and GPT-3.5. Its design allows deployment on phones, and it can achieve a processing speed of 12 tokens per second on the A16 chip used in iPhone 14 Pro and iPhone 15.
- phi-3-small: This model has 700 million parameters. It uses the tiktoken tokenizer to support multiple languages and additionally adds 10% multilingual data. phi-3-small scored 75.3% on the MMLU test, performing excellently and surpassing Meta's recently releasedLlama 3 8B Instruct model.
- phi-3-medium: A medium-sized model with 1.4 billion parameters, trained on more data, and surpassing GPT-3.5 and Mixtral 8x7b MoE in most tests. This model scored 78.2% on the MMLU test, demonstrating its powerful language processing capability.
Phi-3's training method
- High-quality datasets: The training of the Phi-3 model used large-scale datasets of 3.3 trillion to 4.8 trillion tokens. These datasets underwent strict screening and filtering to ensure the educational level and quality of the data.
- Synthetic data generation: Large language models (LLMs) are used to generate synthetic data, which is used to teach the model logical reasoning and various professional skills.
- Staged training: The training process is divided into two stages. The first stage mainly uses web data, with the aim of enabling the model to master general knowledge and language understanding; the second stage further filters web data and combines it with synthetic data for training.
- Data optimization: The training data is calibrated to approach a "data optimal" state, which means that in data selection, priority is given to web data that can improve the model's reasoning ability.
- Post-training optimization: After pre-training, the model also underwent supervised instruction fine-tuning and preference adjustment (DPO), as well as red teaming and automated testing, to improve the model's safety, robustness, and adaptability to dialogue formats.
- Safety and alignment: The development of Phi-3-mini followed Microsoft's responsible AI principles, through safety alignment in the post-training stage, training with helpfulness and harmlessness preference datasets, and iterative review by an independent red team to further identify areas for improvement.
- Quantization processing: To enable the model to run on phones, Phi-3-Mini can be quantized to 4-bit, significantly reducing the model's memory usage.
- Multilingual support: Although Phi-3-Mini is primarily aimed at English, Microsoft is also exploring the multilingual capabilities of small language models, and the Phi-3-Small model was trained by including more multilingual data.
Phi-3 Performance
- Benchmark Scores：phi-3-mini It scored 69% on the MMLU (Massive Multitask Language Understanding) benchmark and 8.38 on MT-bench.phi-3-small and phi-3-medium achieved scores of 75.3% and 78.2% on MMLU, respectively, showing the performance improvement brought by the growth in model size.
- Performance Comparison with Other Models：The Phi-3 series is comparable in performance to models with larger parameter counts such as Mixtral 8x7B and GPT-3.5. In particular,phi-3-mini, despite having only 380 million parameters, surpassed the 8B-parameter Llama 3 in multiple benchmark tests.In comparison with Meta's Llama-3-8B-Instruct model,phi-3-small(7B parameters) scored higher on MMLU (75.3% vs 66%), demonstrating the performance advantage of the Phi-3 series at a smaller model size.
- Running on Mobile Devices：phi-3-mini was specifically designed to run on mobile devices. After 4-bit quantization, it can achieve a speed of 12 tokens per second on the A16 chip used in the iPhone 14 Pro and iPhone 15, which was difficult to achieve with previous large models.
- Multilingual Capabilities：phi-3-small By using the tiktoken tokenizer and adding 10% multilingual data, the model's multilingual processing capability was enhanced, which is an important advantage in comparisons.
Phi-3's Limitations and Shortcomings
- Knowledge Storage Limitations: Due to model size limitations, especially for phi-3-mini, its ability to store large amounts of factual knowledge is limited, which is particularly evident in tests requiring extensive background knowledge, such as its low score on the TriviaQA test.
- Dependence on Search Functionality: To alleviate the limitations of knowledge storage, Microsoft proposed that the model's performance can be enhanced through integration with search engines. This indicates that without external search support, the model may not be able to independently provide the best answers or information.
- Limitations of Multilingual Capabilities: Although phi-3-small was optimized by adding multilingual data, the primary language capability of the Phi-3 series models is still mainly focused on English. This may be a limitation for application scenarios requiring extensive multilingual support.
- Performance on Specific Tasks: On certain specific tasks or datasets, the Phi-3 model may not yet have achieved optimal performance, which may require further training data optimization and model adjustment.
- Security Challenges: Although the model has been optimized for safety, like most large language models, the Phi-3 series models may still face challenges such as hallucinations, inappropriate content, amplification of bias, or security issues.
- Hardware Deployment Limitations: Although Phi-3-Mini can run on smartphones, for some tasks requiring extremely high computing resources, even miniaturized models may face limitations in hardware performance.