What is Qwen2.5-1M
Qwen2.5-1M is an open-source model launched by the Alibaba Tongyi Qianwen team, supporting a context length of 1 million Tokens. The models include Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M. It performs excellently in long-context tasks, outperforming the previous 128K version, especially when handling tasks exceeding 64K in length. The Qwen2.5-14B-Instruct-1M model defeated Qwen2.5-Turbo and consistently surpassed GPT-4o-mini on multiple datasets. The Qwen2.5-1M series models perform comparably to their 128K versions on short-text tasks, ensuring that basic capabilities were not affected by the addition of long-sequence processing ability.
Main features of Qwen2.5-1M
- Long-context task processing: The Qwen2.5-1M series models can handle contexts up to 1 million Tokens, outperforming the previous 128K version. In the Passkey Retrieval task with a context length of 1 million Tokens, the Qwen2.5-1M series models can accurately retrieve hidden information from documents of 1M length.
- Performance advantages: The Qwen2.5-14B-Instruct-1M model defeated Qwen2.5-Turbo and consistently surpassed GPT-4o-mini on multiple datasets.
- Short-sequence task processing: The Qwen2.5-1M series models perform comparably to their 128K versions on short-text tasks, ensuring that basic capabilities were not affected by the addition of long-sequence processing ability.
Technical principles of Qwen2.5-1M
- Long-context training: Qwen2.5-1M adopts a method of gradually extending the length, expanding the context length from 4K to 256K. The training process is divided into multiple stages:
- Pre-training stage: Starting from 4K and gradually increasing to 256K, while using the Adjusted Base Frequency scheme to increase the RoPE base frequency from 10,000 to 10,000,000.
- Supervised fine-tuning stage: Conducted in two stages. The first stage fine-tunes only on short instructions (up to 32K in length), and the second stage mixes short instructions and long instructions (up to 256K) for training.
- Reinforcement learning stage: Training on short text (up to 8K in length). Even when trained on short text, it can generalize human preference alignment performance well to long-context tasks.
- Sparse attention mechanism: To accelerate the prefill stage, Qwen2.5-1M introduced sparse attention optimization based on MInference and proposed a series of improvements:
- Chunked prefill: Divide the input sequence into chunks of length 32768 and perform prefill chunk by chunk, significantly reducing memory requirements.
- Integrated length extrapolation scheme: Integrate the DCA-based length extrapolation scheme into the sparse attention mechanism to improve inference efficiency and the accuracy of long-sequence tasks.
- Sparsity optimization: Propose a method for optimizing the sparsification configuration on sequences of 1 million length, reducing the accuracy loss caused by sparse attention.
- Other optimizations: Optimize operator efficiency and dynamic chunked pipeline parallelism, enhancing the potential of the entire framework.
- Length extrapolation: To extend the model's context length from 256K to 1M, Qwen2.5-1M adopts length extrapolation technology. Through the Dual Chunk Attention (DCA) method, overly large relative positions are remapped to smaller values, solving the performance degradation problem in long-context tasks.
Project address of Qwen2.5-1M
- Project official website：https://qwenlm.github.io/zh/blog/qwen2.5-1m/
- HuggingFace Model Hub：https://huggingface.co/spaces/Qwen/Qwen2.5-1M-Demo
- Technical Papers：https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-1M/Qwen2_5_1M_Technical_Report.pdf
Application Scenarios of Qwen2.5-1M
- In-depth understanding of long novels: Can process 10 long novels at once, conducting in-depth analysis and understanding of long novels.
- Reading multiple papers: Can process multiple academic papers simultaneously, helping researchers quickly obtain key information.
- Literary creation: Assists writers in creating long novels, providing writing inspiration and creative content.
- Advertising copywriting: Helps advertising professionals quickly generate attractive advertising copy.
- Teaching assistance: Serves as a teaching assistance tool in the education field, helping students understand complex concepts.
- Data analysis: In the research field, helps analyze large amounts of data and literature.