MiniMax-01 - MiniMax's new open-source series of models

What is MiniMax-01

MiniMax-01 is a new series of models launched by MiniMax, including the foundational large language model MiniMax-Text-01 and the visual multimodal large model MiniMax-VL-01. MiniMax-01 is the first to implement a linear attention mechanism on a large scale, breaking the limitations of the traditional Transformer architecture, with 456 billion parameters, 45.9 billion activated per pass, performance comparable to top overseas models, and the ability to efficiently handle the world's longest 4 million token context. The MiniMax-01 series models provide API services with extreme cost-effectiveness, low standard pricing, and excellent performance in long-text tasks, multimodal understanding, and many other areas.

MiniMax-01's performance

• Parameter count and activation count: The model has up to 456 billion parameters, with 45.9 billion activated per pass, and overall performance comparable to top overseas models.
• Ultra-long context processing capability: It can efficiently handle the world's longest 4 million token context, 32 times that of GPT-4o and 20 times that of Claude-3.5-Sonnet.
• Mainstream benchmark results: It matches the two overseas models widely recognized as the most advanced, GPT-4o-1120 and Claude-3.5-Sonnet-1022, on most tasks.
• Advantages in long-text tasks: On long-text tasks, its performance degrades the slowest, significantly outperforming Google's Gemini model.
• Efficiency in processing long inputs: It approaches linear complexity and has a clear advantage compared with other top global models.

MiniMax-01's main functions

• Language understanding and generation：
  • Text summarization: It can extract key information from long articles and generate concise and accurate summaries.
  • Translation: It enables accurate conversion between different languages while maintaining semantic completeness and accuracy.
  • Question answering: It answers questions based on given text or its own knowledge. Closed question answering targets specific text content, while open-domain question answering covers a broader range of knowledge.
• Multimodal understanding：
  • Image-text matching: It determines whether the text accurately describes the image content, for scenarios such as image annotation and content moderation.
  • Image description generation: It generates fluent and accurate descriptive text based on images, helping to understand the elements and layout relationships in the image.
  • Visual question answering: It answers questions involving visual content by combining image information.
• Long context processing: It efficiently handles contexts up to 4 million tokens, meeting practical needs such as reading professional books, assisting with programming projects, and analyzing long documents, and providing foundational capabilities for building complex Agent systems.

MiniMax-01's technical principles

•  Linear attention mechanism
  • Core idea: It transforms the computationally heavy self-attention mechanism in the traditional Transformer into linear complexity based on mathematical techniques, allowing the model to efficiently process long sequences.
  • Lightning Attention: An optimized implementation of linear attention that uses chunking techniques to divide attention computation into intra-chunk and inter-chunk computation, using left product within chunks and right product between chunks, maintaining the overall linear computational complexity and improving the efficiency of processing long sequences.
• Hybrid architecture
  • Architecture design: In the model, 7 out of every 8 layers use linear attention, and 1 layer uses traditional SoftMax attention. It combines the efficiency of linear attention with the advantages of SoftMax attention, performing especially well when handling long-context tasks.
  • Performance Validation: Demonstrates the performance and efficiency advantages of the hybrid architecture in long-context tasks. For example, in long-context retrieval tasks, as the input length increases, MiniMax-Text-01 exhibits the slowest performance degradation.
• Model Optimization and Training
  • MoE Optimization: To reduce communication overhead in the MoE architecture, a token grouping overlap scheme is introduced, overlapping communication with token processing of different expert groups to reduce waiting time. An EP-ETP overlap strategy is introduced to improve resource utilization, reduce communication overhead, and enhance training efficiency.
  • Long-Context Optimization: Adopts a "data packing" technique to reduce computational waste. Designs the Varlen Ring Attention algorithm to directly apply ring attention computation to packed sequences, avoiding excessive padding. Proposes the LASP+ algorithm to optimize the parallel computation process of linear attention sequences, achieving fully parallelized computation and improving system efficiency.
  • lightning attention inference optimization: Based on strategies such as batch kernel fusion, separated prefill and decode execution, multi-level padding, and strided batched matmul extension, it reduces memory access operations and improves inference speed.

MiniMax-01 Project Address

• Project Official Website：
  • MiniMax Open Platform：https://www.minimaxi.com
  • MiniMax Open Platform Overseas Edition：https://www.minimaxi.com/en
• GitHub Repository：https://github.com/MiniMax-AI
• Technical Paper：https://filecdn.minimax.chat/_Arxiv_MiniMax_01

MiniMax-01 API Pricing

• MiniMax-Text-01: Powerful long-text processing capability, context length 1000k, input 0.001 yuan/thousand tokens, output 0.008 yuan/thousand tokens.
• MiniMax-VL-01: Powerful visual understanding capability, context length 1000k, input 0.001 yuan/thousand tokens, output 0.008 yuan/thousand tokens.

MiniMax-01 Application Scenarios

• Enterprise Users: Covers content creators, marketing personnel, customer service teams, technical teams, and knowledge management personnel, improving the efficiency of content creation, marketing effectiveness, customer satisfaction, project development, and knowledge sharing.
• Educators and Students: Teachers generate teaching materials, and students use it for learning assistance, improving teaching and learning quality.
• Creative Workers: Writers, poets, lyricists, designers, and artists obtain creative inspiration, assist creative writing and artistic design, and stimulate creative potential.
• Researchers and Scholars: Suitable for processing academic papers and conducting literature reviews, improving the efficiency and depth of scientific research work.
• Developers and Engineers: Natural language processing developers, multimodal application developers, and system integration engineers develop customized applications and enhance system intelligence.