Chitu is a high-performance large-model inference framework focused on efficiency, flexibility, and usability, positioned as a "production-grade large-model inference engine." The project fully considers the progressive needs of enterprise AI implementation, from small-scale trials to large-scale deployment.
Main features:
- Diverse compute adaptation: It not only supports multiple series of NVIDIA products from the latest flagship to older models, but also provides optimized support for domestic chips.
- Full-scenario scalability: From pure CPU deployment, single-GPU deployment, to large-scale cluster deployment, the Chitu engine provides scalable solutions.
- Long-term stable operation: It can be applied in actual production environments, with stability sufficient to handle concurrent business traffic.
Version evolution (as listed in the materials):
- 2026/07/02 released v0.6.0, adding the chitu.run executable: a single file can launch complex tasks such as multi-node, multi-instance, and PD separation.
- 2026/02/06 released v0.5.1, adapting to Moore Threads GPUs.
- 2025/12/12 released v0.5.0, focusing on improving performance in cluster deployment scenarios.
- 2025/08/01 released v0.4.0, greatly improving the performance and stability of all-in-one inference deployment scenarios, adapting to Ascend, NVIDIA, MetaX, and Hygon, and supporting models such as DeepSeek, Qwen, GLM, and Kimi.
- 2025/07/28 released v0.3.9, first supporting Huawei Ascend 910B inference deployment of the Zhipu GLM-4.5 MoE model.
- 2025/06/12 released v0.3.5, providing complete native support for Ascend 910B and providing a high-performance inference solution for the Qwen3 series models.
- 2025/04/29 released v0.3.0, adding efficient operator implementations for online FP4 to FP8 and BF16 conversion, and supporting the FP4 quantized version of DeepSeek-R1 671B.
- 2025/04/18 released v0.2.2, adding support for CPU+GPU heterogeneous hybrid inference, enabling single-card inference of DeepSeek-R1 671B.
- 2025/03/14 released v0.1.0, supporting DeepSeek-R1 671B and providing an efficient operator implementation for online FP8 to BF16 conversion.
Usage: It is recommended to use chitu.run for deployment; please download it from the Assets page in Releases. Please refer to the development manual for complete installation and usage instructions. For more models, please see Supported Models.
Performance data: Please refer to the performance data tested by the development team, and you are also welcome to share your own test data. Performance data is related to hardware configuration, software version, and test load, and results from multiple tests may fluctuate.
Community and support: The project welcomes participation from the open-source community in co-building; please refer to the contribution guide. If you have questions or concerns, you are welcome to submit an issue. If you need professional technical services, you can email solution@chitu.ai.
License: This project uses the Apache License v2.0 license; see the LICENSE file for details. The repository also references some code snippets from other open-source projects, and the relevant copyright information has been marked in the code in SPDX format; license information can be found in the LICENSES/ directory. The repository also contains third-party submodules that follow other open-source licenses, located in the third_party/ directory. The project thanks Huawei, MetaX, Hygon, Enflame, Zhipu, China Telecom, Parallel Technology, and others for their help, and has learned from and reused some functions from multiple open-source projects (sorted alphabetically).
