NanoFlow - A serving framework for optimizing large language model inference throughput

What is NanoFlow

NanoFlow is a high-performance serving framework designed specifically for large language models (LLMs) to improve model inference throughput. It optimizes the model inference process by using compute, memory, and network resources in parallel within a single device. The parallel processing mechanism enables NanoFlow to handle more requests simultaneously while ensuring fast responses, thereby significantly improving the overall system performance and user experience.

Main features of NanoFlow

• Improved inference throughput: The core goal of NanoFlow is to maximize model inference throughput, that is, to increase the number of tokens processed per second while ensuring reasonable latency.
• Intra-device parallelism: Through operation-level pipelining and execution unit scheduling, NanoFlow can process different operations in parallel within a single device, thereby improving resource utilization.
• Automated parameter search: NanoFlow uses an automated parameter search algorithm to adapt to different models, reducing manual intervention and simplifying the model deployment and optimization process.
• Global batch scheduling: Requests are managed through a global batch scheduler, which selects the optimal batch size to improve computational efficiency.
• Operation-level parallelism engine: Splits requests into smaller batches (nano-batches) and assigns them to different execution units to achieve operation-level parallelism.

Technical principles of NanoFlow

• Global batch scheduler: Improves computational efficiency by managing requests and selecting the optimal dense batch size.
• Intra-device parallelism engine: Splits requests into smaller batches (called nano-batches) and assigns them to different execution units to achieve operation-level parallelism.
• KV cache manager: Optimizes memory usage by predicting peak memory usage and promptly offloading the KV cache of completed requests to lower-tier storage.

NanoFlow project address

• GitHub repository address：https://github.com/efeslab/Nanoflow
• arXiv technical paper：https://arxiv.org/pdf/2408.12757

How to use NanoFlow

• Visit the GitHub repository: Go to the GitHub repository to obtain the latest version of NanoFlow and related documentation.
• Read the documentation: In the GitHub repository, check the README file and related documentation.
• Install the framework: Install using specific commands or through a package manager.
• Run examples: Run the example code to ensure that NanoFlow works properly.
• Customize and extend: Customize and extend NanoFlow according to your needs.

Application scenarios of NanoFlow