Xorbits Inference (Xinference for short) is a powerful and versatile model inference and serving library, designed for deploying and running language models, speech recognition models, and multimodal models. With a single command, you can deploy and run your own models or built-in state-of-the-art models. The library is aimed at researchers, developers, and data scientists, providing a unified, production-oriented inference API.
Main capabilities
- Simplified model serving: deploy and serve large language, speech recognition, and multimodal models with a single command, for experimentation and production environments.
- Built-in state-of-the-art models: provides access to cutting-edge open-source models. The built-in supported models listed in the material include EmbeddingGemma 2, Confucius4-TTS, jina-ocr-v1, TeleOCR, the Ming-Image series (Design, Design Layer), Qwen-Image-2.1, MinerU2.5, the Spark-X2.5 series, the LingBot-World-V2 series, the Irodori-TTS v4.1 series, YuE2-3B, the AuK series, and more.
- Heterogeneous hardware utilization: leverages ggml to intelligently use heterogeneous hardware such as GPUs and CPUs to accelerate inference.
- Flexible APIs and interfaces: supports an OpenAI-compatible RESTful API (including the Function Calling API), RPC, CLI, and WebUI.
- Distributed deployment: supports distributing model inference across multiple devices or machines.
- Third-party integrations: integrates with libraries such as LangChain, LlamaIndex, Dify, and Chatbox.
- Framework enhancements: Xinference 4.0.0 provides the multi-engine Xavier, heterogeneous P/D, and runtime model reconfiguration; features Agent-native serving (integrated with Xagent, supporting dynamic planning, tool calling, and autonomous multi-step reasoning); automatic batching (automatically batching concurrent requests to improve throughput); Xllamacpp (llama.cpp Python bindings maintained by the Xinference team, supporting continuous batching); distributed inference (running models across workers); and VLLM enhancements (sharing the KV cache across multiple replicas).
Usage
- Self-hosted community edition: you can run Xinference in your environment via the quick start guide.
- Enterprise edition: Xinference Enterprise is the commercial version for running models in production environments, and can be deployed in your own cloud or data center, or you can choose Xinference Cloud's managed dedicated deployment; the hosted Model API allows access to supported models via an OpenAI-compatible API, without managing inference infrastructure. For enterprise inquiries, visit xinference.co or send an email.
- One-click installation: Windows PowerShell provides a one-click installer; Nvidia GPU users can use the Xinference Docker image to start the server (Docker and CUDA must be configured first); Kubernetes clusters need GPU support and then can be installed according to the documentation.
- The list of built-in models and deployment instructions can be obtained through the documentation.
Applicable scenarios
- Running open-source models, speech models, and multimodal models in the cloud, on-premises, or on a personal laptop.
- Model serving scenarios that require a unified, production-ready inference API.
- Use in combination with platforms or libraries such as LangChain, LlamaIndex, Dify, Chatbox, Xagent, FastGPT, RAGFlow, and MaxKB.
- Distributed inference and heterogeneous hardware acceleration scenarios.
Limitations and information not yet provided The material does not provide specific pricing, performance benchmark data, precise minimum hardware requirements, or the specific license terms of each model. The specific code content of some installation commands is not fully given in the provided material; for details, refer to the official documentation.
