What is xLLM
xLLM is an efficient intelligent inference framework open-sourced by JD, specifically optimized for domestic chips, supporting integrated device-cloud deployment. The framework uses a service-engine separation architecture, with the service layer responsible for request scheduling and fault tolerance, and the engine layer focused on computation optimization. It features multi-stream parallelism, graph fusion, and dynamic load balancing. xLLM supports various scenarios such as large models, multimodal models, and generative recommendation, providing high-performance, low-cost inference services to help businesses such as intelligent customer service, real-time recommendation, and content generation land efficiently, and promoting the large-scale application of large language models on domestic chips.
Main Features of xLLM
- Fully graph-based / multi-layer pipeline execution orchestration: Through framework-layer asynchronous decoupled scheduling, model graph-layer computation-communication asynchronous parallelism, and operator kernel-layer deep pipeline optimization, it achieves multi-layer pipeline execution orchestration, reducing computation bubbles and improving overall inference efficiency.
- Graph execution optimization for dynamic shapes: It uses parameterization and multi-graph caching methods to achieve dynamic size adaptation, combined with a controlled memory pool and custom operator integration, improving static graph flexibility and ensuring safe memory reuse, and optimizing dynamic input processing performance.
- MoE operator optimization: For MoE models, it implements GroupMatmul and Chunked Prefill operator optimizations, respectively improving computational efficiency and the processing capability for long-sequence inputs, and enhancing model inference performance.
- Efficient memory optimization: It adopts mapping management between discrete physical memory and contiguous virtual memory, allocates memory space on demand, intelligently schedules memory page reuse, reduces memory fragmentation and allocation latency, adapts to domestic chip operators, and improves memory utilization efficiency.
- Global multi-level KV Cache management: It implements intelligent KV offloading and prefetching for multi-level caches, builds a distributed storage architecture centered on KV Cache, optimizes intelligent transmission routing of KV across multiple nodes, and improves cache efficiency and data transmission performance.
- Algorithm optimization: Through speculative inference optimization and dynamic load balancing of MoE experts, it achieves multi-core parallelism to improve efficiency, dynamically adjusts expert distribution, optimizes algorithm performance, and improves inference throughput and load balancing capability.
How to use xLLM
- Environment preparation：
- Download image: According to the hardware device (such as A2, A3, etc.) and architecture (x86 or arm), select the appropriate Docker image. For example, for A2 devices (x86 architecture), you can download the
xllm/xllm-ai:0.6.0-dev-hb-rc2-x86image. If the download fails, try the backup sourcequay.io/jd_xllm/xllm-ai:0.6.0-dev-hb-rc2-x86。 - Create container: When creating and starting the container, you need to mount the necessary devices and directories to ensure the container can access hardware resources and data. This includes device files (such as
/dev/davinci0、/dev/davinci_manager, etc.), model file paths, driver paths, etc.
- Download image: According to the hardware device (such as A2, A3, etc.) and architecture (x86 or arm), select the appropriate Docker image. For example, for A2 devices (x86 architecture), you can download the
- Installation and compilation：
- Install dependencies：
- Clone repository: After entering the container, clone the official xLLM repository and initialize the submodules.
- Configure vcpkg: If vcpkg is not preinstalled in the image, manually clone the vcpkg repository and set the environment variable
VCPKG_ROOTto point to the vcpkg installation path. - Install Python dependencies: Use Tsinghua University's Python mirror source to install the Python dependencies required by xLLM, upgrade
setuptoolsandwheel。
- Compile：
- Compile and generate executable files: Run the compilation command to generate an executable file. By default, the compilation target is the A2 device. If you need to compile for other devices (such as A3 or MLU), specify the device type by adding parameters.
- Generate whl package: If you need to generate a Python whl package, run the corresponding compilation command. The generated whl package will be saved in the
dist/directory.
- Install dependencies：
- Model loading：
- Prepare model files: Prepare the model files and place them in a path accessible to the container, for example
/mnt/cfs/9n-das-admin/llm_models。 - Load model: Use the interface provided by xLLM to load the model. After the model is loaded, inference tasks can be performed.
- Prepare model files: Prepare the model files and place them in a path accessible to the container, for example
- Inference call: Through the inference interface provided by xLLM, input text or other data to obtain inference results. The inference interface performs calculations based on the loaded model and returns the results.
xLLM project address
- Project official website：https://xllm.readthedocs.io/
- GitHub repository：https://github.com/jd-opensource
xLLM application scenarios
- Intelligent customer service: Quickly respond to user inquiries, provide accurate answers and suggestions, and improve customer satisfaction and customer service efficiency.
- Real-time recommendation: Based on user behavior data, generate personalized recommendation content in real time to improve user engagement and conversion rates.
- Content generation: xLLM can generate high-quality text content, such as news, articles, creative copy, etc., assisting content creation.
- Multimodal applications: Supports multimodal models (such as text + image), used in scenarios such as image description generation and visual question answering.
- Generative recommendation: Combines generative technology to generate richer and more personalized recommendation results, improving user experience.