DeepSpeed-MII (Model Implementations for Inference) is an open-source Python library launched by the DeepSpeed team, with the goal of making powerful model inference easier to use and focusing on optimizing high throughput, low latency, and cost efficiency.
Purpose and Positioning MII is used to accelerate model inference and is powered at the underlying level by DeepSpeed-Inference. Based on the model architecture, model size, batch size, and available hardware resources, it automatically applies the appropriate combination of system optimizations to minimize latency and maximize throughput.
Main Capabilities MII provides four key technologies for high-throughput text generation: Blocked KV Caching, Continuous Batching, Dynamic SplitFuse, and High Performance CUDA Kernels. Previously, MII also introduced related optimizations for low-latency serving scenarios, including DeepFusion for Transformers, multi-GPU inference based on Tensor-Slicing, ZeRO-Inference for resource-constrained systems, and compiler optimizations.
Model Support MII currently supports more than 37,000 models across eight mainstream model architectures, and plans to add more models in the near future. All current models rely on Hugging Face at the backend to provide model weights and the corresponding tokenizer. MII Legacy APIs support more than 50,000 different models, including BERT, RoBERTa, Stable Diffusion, and text generation models such as Bloom and GPT-J.
Usage Users can quickly get started through the PyPI release of DeepSpeed-MII, using either a non-persistent pipeline or a persistent deployment. A non-persistent pipeline exists only during the currently running Python script, and the complete deployment example is only 4 lines of code; its call returns a list of Response objects, which provide access to fields such as generated_text, prompt_length, generated_length, and finish_reason, and the pipeline can be destroyed via the destroy method to release device memory. Persistentdeployment is suitable for long-running and production applications. It uses a lightweight GRPC server and can be queried by multiple clients at the same time. The complete example is 5 lines of code, and the server can be shut down through any client.
Parallelism and Scaling On multi-GPU systems, when using the deepspeed launcher, tensor parallelism is automatically controlled by the --num_gpus flag. Persistent deployment supports model parallelism through the tensor_parallel parameter of mii.serve, and multiple model replicas can also be set up with load balancing provided by DeepSpeed-MII. Model parallelism and replicas can be combined to accommodate systems with more GPUs. The choice between model parallelism and model replicas depends on the hardware, model, and workload characteristics: for small models, using model replicas may achieve the lowest average request latency, while for large models, using model parallelism may achieve higher overall throughput.
RESTful API and Deployment Options When creating a persistent MII deployment, enable_restful_api=True can be set to enable the RESTful API, and the request address is http://{HOST}:{RESTFUL_API_PORT}/mii/{DEPLOYMENT_NAME}。mii.serve() Provides options such as model_name_or_path (required), max_length, deployment_name, tensor_parallel, replica_num, enable_restful_api, restful_api_port, etc.; mii.client() obtains a client via the model name or deployment_name. Generation parameters can be controlled for a single prompt, such as max_length, min_new_tokens, max_new_tokens, ignore_eos, top_p, top_k, temperature, do_sample, return_full_text.
Installation and Dependencies The fastest way to get started is to install via the PyPI version of DeepSpeed-MII. To facilitate use and reduce compilation time, the project provides precompiled Python wheels via DeepSpeed-Kernels that cover most custom kernels. This library has good portability in NVIDIA GPU (compute capabilities 8.0+, i.e., Ampere+), CUDA 11.6+, and Ubuntu 20+ environments, and is usually installed automatically as a dependency of DeepSpeed-MII; if you need to manually compile the kernels, you can refer to the advanced installation documentation.
Limitations and Notes Regarding specific charges, pricing, platform support scope, performance guarantees, and other information, the materials do not currently provide it.
