What is Chitu
Chitu is a high-performance large model inference engine jointly open-sourced by the High Performance Computing Institute of Tsinghua University and Qingcheng Jizhi. It is designed specifically to address the high cost and low efficiency issues of large models during the inference stage. It has strong hardware adaptation capabilities, supports multiple NVIDIA GPUs and domestic chips, and breaks the dependence on specific hardware (such as NVIDIA Hopper architecture). In terms of performance, when deployingDeepSeek-R1-671B on an A800 cluster, compared with some foreign open-source frameworks, GPU usage is reduced by 50% and inference speed is increased by 3.15 times. It supports full-scenario deployment from pure CPU to large-scale clusters, and can meet inference needs at different scales and scenarios.
Main features of Chitu
- Diverse computing power adaptation: Supports multiple series of NVIDIA GPUs from the latest flagship to older models, while also providing optimized support for domestic chips, breaking the dependence on NVIDIA Hopper architecture.
- Full-scenario scalability: From pure CPU deployment, single-GPU deployment to large-scale cluster deployment, the Chitu engine can provide scalable solutions to meet inference needs at different scales and scenarios.
- Low-latency optimization: For latency-sensitive scenarios, such as financial risk control, it optimizes model inference speed and reduces response time.
- High-throughput optimization: In high-concurrency scenarios, such as intelligent customer service, it increases the number of requests processed per unit time.
- Low-VRAM optimization: Reduces VRAM usage per card, enabling enterprises to achieve higher inference performance with fewer hardware resources.
- Long-term stable operation: The Chitu engine can be applied in actual production environments, and its stability is sufficient to handle concurrent business traffic.
- Out-of-the-box: Qingcheng Jizhi has launched an inference all-in-one machine based on Chitu, providing out-of-the-box deployment solutions and professional operations and maintenance services, further simplifying the process of enterprise AI implementation.
Technical principles of Chitu
- Underlying technological innovation: Through underlying technological innovation, the Chitu engine has for the first time achieved native operation of FP8 precision models on non-NVIDIA Hopper architecture GPUs and various domestic chips. It breaks the past dependence of FP8 models on NVIDIA H-series high-end GPUs, enabling enterprises to efficiently deploy large models on more types of hardware.
- Operator-level optimization: Chitu has carried out instruction-level optimization for key operators (such as GeMM, MoE, etc.), directly processing FP8 data rather than simple quantization. This ensures lossless model precision while significantly improving inference speed. In tests on an A800 cluster, compared with some foreign open-source frameworks, the Chitu engine still achieved a 3.15x improvement in inference speed while reducing GPU usage by 50%.
- Full-scenario performance optimization: The Chitu engine supports low latency, high throughput, and low VRAM optimization, and can provide optimal solutions under different hardware configurations and system environments according to different scenario requirements. Chitu can find the best balance among GPU utilization, memory efficiency, and network transmission according to system resource conditions.
- Parallel computing and compilation optimization: The Chitu engine condenses many years of technical accumulation in parallel computing and compilation optimization by the Tsinghua University team. Through intelligent compilation technology, the team can accelerate the development process of high-performance operators and achieve optimization for different hardware architectures in a relatively short time.
Chitu project address
- Github repository：https://github.com/thu-pacman/chitu
Application scenarios of Chitu
- Risk Identification and Early Warning: The Chitu Engine can quickly process massive transaction data, monitor potential risks in real time and issue timely warnings, helping financial institutions improve risk management efficiency.
- Intelligent Customer Service and Customer Experience Optimization: Through a large-model intelligent knowledge base, it quickly responds to customer needs and improves the service experience.
- Disease Diagnosis Assistance: The Chitu Engine can quickly process medical data, improve the speed and accuracy of disease diagnosis, and enhance patients' medical experience.
- Traffic Flow Optimization: The Chitu Engine can process traffic data in real time, optimize traffic flow, and alleviate urban congestion.
- Scientific Research Data Analysis: The Chitu Engine can efficiently process scientific research data and accelerate the research process.