Mooncake - The large model inference architecture launched by Moonshot AI's Kimi in collaboration with Tsinghua University and other institutions

What is Mooncake

Mooncake is an open-source large model inference architecture jointly developed by Moonshot AIKimiin collaboration with Tsinghua University and other institutions. It adopts a KVCache-centric distributed architecture, separating prefill and decode clusters, and fully utilizes the underutilized CPU, DRAM, and SSD resources in GPU clusters to achieve efficient KVCache caching. The core advantage of Mooncake lies in its ability to significantly improve the throughput of large model inference, reduce computational overhead, and handle high-load scenarios while maintaining service-level objectives (SLOs) related to service latency. The architecture performs excellently in long-context scenarios, significantly improving throughput, while supporting a prediction-based early rejection strategy to optimize resource allocation under overload conditions. The Mooncake project is open-sourced on GitHub, promoting the development of an efficient inference platform for large model technology.

Main Features of Mooncake

• Efficient large model inference: Through its distributed architecture, Mooncake optimizes the inference process of large models, especially when processing long-context data, significantly improving inference throughput.
• KVCache-centric design: With KVCache at its center, Mooncake achieves efficient data caching and reuse, reducing dependence on GPU resources and lowering computational overhead.
• Separation of prefill and decode: The architecture separates the prefill and decode stages, allowing resources to be optimized for the computational characteristics of different stages.
• Resource optimization: Through its disaggregated design, Mooncake can more effectively utilize CPU, DRAM, and SSD resources, improving resource utilization.
• Load balancing: Mooncake implements a balancing strategy based on cache load, and through an automatic hotspot migration scheme, improves cache hit rate and system load balancing.
• Overload management: In the face of high load, Mooncake adopts a prediction-based early rejection strategy to optimize resource allocation and reduce ineffective computation.
• High-performance transmission: Based on RDMA technology, Mooncake achieves high-speed cross-node KVCache transmission, reducing latency.
• Standardized interface: Mooncake creates a new standard interface for high-performance memory-semantic storage for the large model era, providing a reference implementation.
• Cost reduction: By optimizing the inference process and resource utilization, Mooncake helps reduce the cost of large model inference, making AI technology more economical and efficient.

Technical Principles of Mooncake

• Distributed architecture: Utilizes CPU, DRAM, and SSD resources in GPU clusters to achieve distributed storage and transmission of KVCache, increasing cache capacity and transmission bandwidth, and reducing dependence on a single GPU resource.
• Global scheduler (Conductor): Responsible for scheduling requests based on the current KVCache distribution and workload, and deciding on the replication or swapping of KVCache blocks, optimizing overall throughput and meeting service-level objectives (SLOs).
• Chunked Pipeline Parallelism: For long-context requests, input tokens are divided into multiple chunks and processed in parallel on different nodes to reduce latency.
• Layer-wise prefill: Asynchronously loads and stores KVCache, reducing VRAM usage by overlapping transmission and computation.
• Cache-aware scheduling:Mooncake's scheduling algorithm takes into account KVCache reuse, prefill time, and instance load queuing time to achieve efficient request scheduling.

Mooncake's project address

• Github repository：https://github.com/kvcache-ai/Mooncake
• arXiv technical paper：https://arxiv.org/pdf/2407.00079

Mooncake's application scenarios

• Natural Language Processing (NLP):Mooncake can be used to support various NLP tasks, such as language translation, text summarization, sentiment analysis, question-answering systems, and chatbots, among others.
• Content recommendation systems:In recommendation systems, Mooncake can be used to process user behavior data and content features to provide personalized recommendations.
• Search engines:Mooncake can be used to improve search engines' query understanding and document ranking, providing more accurate search results by understanding complex query intents and document content.
• Speech recognition and generation:In the field of speech recognition, Mooncake can be used to improve the accuracy of speech-to-text conversion; in the field of speech generation, it can generate more natural and fluent speech output.
• Image and video analysis:Efficient inference capabilities can also assist image and video analysis tasks, such as image annotation, video content understanding, and more.
• Intelligent customer service and virtual assistants:Mooncake can provide powerful backend support, enabling intelligent customer service and virtual assistants to handle complex conversations and tasks.