KTransformers is a research project focused on efficient inference and fine-tuning of large language models (LLMs), achieving cutting-edge inference and fine-tuning optimization through CPU-GPU heterogeneous computing. The project is developed and maintained by the MADSys Lab at Tsinghua University, and community contributions are welcome. The current project provides two major user capabilities from the kt-kernel source tree: Inference and SFT fine-tuning. The original integrated KTransformers framework has been archived in the archive/ directory to allow clearer maintenance of documentation and code.
Inference — High-performance kt-kernel service
KTransformers' inference capability targets heterogeneous LLM inference and provides CPU-optimized kernel operations. Main capabilities include:
- AMX/AVX acceleration: Provides Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference.
- MoE optimization: Achieves efficient Mixture-of-Experts inference through NUMA-aware memory management.
- Quantization support: INT4/INT8 quantized weights on the CPU side, with GPTQ support on the GPU side.
- Easy integration: Provides a concise Python API that can be integrated into SGLang and other frameworks.
Applicable scenarios include: CPU-GPU hybrid inference for running large MoE models on consumer-grade hardware; integration with SGLang for production serving; heterogeneous expert placement (hot experts on GPU, cold experts on CPU).
SFT fine-tuning — Combined with LlamaFactory
KTransformers is integrated with LlamaFactory for fine-tuning ultra-large-scale MoE models. The project provides an easy-to-use fine-tuning Cookbook, covering hardware checks, installation, BF16/FP8/INT8 configurations, LoRA and full training, resource planning, and troubleshooting. Main capabilities include:
- Multi-backend support: CPU/GPU hybrid fine-tuning, with INT8/INT4 quantization support.
- Ultra-large-scale MoE support: Enables fine-tuning of models such as DeepSeek-V3/R1 under limited GPU memory.
- Performance advantages: In the benchmarked MoE SFT workloads, compared with ZeRO-Offload, it delivers 6-12x training speedup; under the benchmark configuration, CPU memory usage is about half that of the previous KT SFT solution.
- LlamaFactory integration: Seamless integration with the popular fine-tuning framework.
Supports Kimi K2.5 / K2.6 LoRA fine-tuning, using native RAWINT4 routed experts; end-to-end validation uses Kimi K2.5, and K2.6 follows the same model architecture path.
Version updates (partial)
The project continues to iterate rapidly. Recent updates include: release of the KTransformers-FineTune (KT-FT) author preprint; Kimi K2.5 / K2.6 LoRA fine-tuning support; native support for GLM-5.3-flash (1M token context and multimodal input, targeting consumer-grade GPUs); KTransformers × LlamaFactory MoE fine-tuning Cookbook; LoRA fine-tuning support compatible with AVX512 x86 CPUs (including AMD servers, no AMX required); DeepSeek-V4-Flash running on a single Ascend NPU via CPU expert offloading; native block-FP8 LoRA fine-tuning; end-to-end BF16 full-parameter fine-tuning for MoE models; Day0 support for MiniMax-M3, GLM-5.2, DeepSeek-V4-Flash, MiniMax-M2.5, GLM-5, Kimi-K2.5, etc.; CPU-GPU expert scheduling, native BF16 and FP8 per-channel precision, and unified fine-tuning and inference on AutoDL; RL-DPO fine-tuning; SGLang integration; Ascend NPU support; Intel Arc GPU support; AMX-Int8, AMX-BF16, and Qwen3MoE; 3-tier (GPU-CPU-Disk) prefix cache reuse, etc.
Usage and documentation
The project provides quick-start entry points for inference and SFT, complete documentation, and an online book. For specific installation steps, commands, code examples, etc., please refer to the documentation and tutorials in the official repository. If you use KTransformers in your research, please cite the relevant papers (the inference paper and the fine-tuning paper; the complete BibTeX is available in the project description).
Applicable scenarios and limitations
KTransformers is suitable for those who wish to Researchers and developers who run or fine-tune large MoE models under limited GPU memory using CPU-GPU heterogeneous approaches, and it is also applicable to scenarios integrated with frameworks such as SGLang and LlamaFactory. Its capabilities focus on inference and fine-tuning optimization; for the specific supported models, hardware, quantization formats, and performance data, please refer to the official documentation and release notes. The project is a research project, and before use it is recommended to consult the official guide to confirm compatibility.
