xllm

xLLM is a high-performance LLM inference engine that supports LLM, VLM, DiT, and REC models, and is optimized for a variety of AI accelerators. It is hosted by the OpenAtom Foundation.

[Purpose] xLLM is an efficient LLM inference framework, specifically optimized for domestic AI accelerators, aimed at enterprise-level deployment, with the goal of improving efficiency and reducing costs.

[Main Capabilities]

• Top-tier performance: Achieves high throughput and low latency inference through multiple advanced features.
• Mainstream hardware support: Specifically built and deeply optimized for domestic AI accelerators.
• Decoupled service and engine architecture: The service layer handles scheduling and availability, while the engine layer handles computation.
• Enterprise-level deployment: Has been claimed to be validated in large-scale scenarios in JD.com's core retail business.
• Supported model types: LLM, VLM, DiT, and REC models.
• Hybrid KV cache management: Built on Mooncake, supports global KV cache management, with intelligent offloading and prefetching capabilities.

[How to Use] The repository provides the following introductory content: hardware support documentation, quick start, offline inference, supported models, community and support. The project description mentions that deployment can be carried out by referring to the deployment documentation. Specific installation commands, code examples, and configuration steps are not currently provided in the supplied materials.

[Version Support Record (as listed in the materials)]

• 2026-08-27: day-0 support for the GLM-5.3-Flash model.
• 2026-07-06: xLLM was officially donated to the OpenAtom Foundation.
• 2026-06-13: day-0 support for the MiniMax-M3 model.
• 2026-04-24: day-0 support for the DeepSeek-V4 model.
• 2026-02-12: day-0 support for high-performance inference of the GLM-5 model.
• 2025-12-21: day-0 support for high-performance inference of the GLM-4.7 model.
• 2025-12-08: day-0 support for high-performance inference of the GLM-4.6V model.
• 2025-12-05: Support for high-performance inference of the GLM-4.5/GLM-4.6 series models; support for high-performance inference of the VLM-R1 model; built Mooncake-based hybrid KV cache management.
• 2025-10-16: Published the xLLM technical report on arXiv.

[Applicable Scenarios] Targeted at enterprise-level scenarios that require inference deployment of LLM, VLM, DiT, and REC models on domestic AI accelerators, especially large-scale production environments with high requirements for throughput, latency, and deployment efficiency.

[Limitations and Information Not Currently Provided] The materials do not provide specific installation commands, code examples, model weight acquisition methods, license agreement details, fee and pricing information, performance benchmark data, or a complete hardware compatibility list. Before use, refer to the deployment documentation and hardware support documentation in the official repository.