AIKit is a comprehensive platform designed to help users quickly get started with hosting, deploying, building, and fine-tuning large language models (LLMs). It provides three main capabilities: inference, fine-tuning, and OCI packaging.
Purpose: AIKit is used to simplify the deployment, building, and fine-tuning process of open-source LLMs, enabling developers to easily run and customize language models locally or in a Kubernetes environment.
Main capabilities:
- Inference: AIKit uses LocalAI and supports a wide range of inference capabilities and formats. LocalAI provides a REST API compatible with the OpenAI API, so any client compatible with the OpenAI API (such as Kubectl AI, Chatbot-UI, etc.) can be used to send requests to open-source LLMs.
- Fine-tuning: AIKit provides a scalable fine-tuning interface and supports Unsloth, delivering a fast, memory-efficient, and easy-to-use fine-tuning experience.
- OCI packaging: Package models as OCI artifacts so they can be distributed through any OCI-compliant registry. Supports the CNCF ModelPack specification and general artifact packaging.
How to use: AIKit does not require a GPU, internet access, or additional tools, only Docker or Podman. It provides declarative configuration for inference and fine-tuning. Users can quickly get started on their local machine, and after running, access http://localhost:8080/chat to use the WebUI. AIKit provides endpoints compatible with the OpenAI API, so any client compatible with the OpenAI API can be used. AIKit also provides preset models that work out of the box. Users can also create their own images and host them in the container registry of their choice.
Applicable scenarios:
- LLM deployment in local or Kubernetes environments.
- Integration with clients compatible with the OpenAI API is required.
- Fine-tune open-source LLMs.
- Distribute models through OCI registries.
- Support multimodal models and image generation.
- Support GGUF (llama) and GGML (llama-ggml) models.
- Support AMD64 and ARM64 CPUs, as well as NVIDIA CUDA and AMD ROCm GPU-accelerated inference.
- Support air-gapped inference using self-hosted or local registries, provided that model content and dependencies are pre-baked or mirrored.
Limitations:
- By default, the runner image that downloads models at startup does not support air-gapped environments.
- AMD ROCm support is currently only available for the llama-cpp backend on linux/amd64 and is experimental.
- Apple Silicon support is experimental, supports only gguf models, and applies only to Apple Silicon, not other architectures (including Intel Mac).
- The preset GPU images are based on CUDA; for AMD GPUs, you need to create your own images.
- For the specific usage steps and configuration details of some features (such as fine-tuning and OCI packaging), please refer to the AIKit website documentation.
Other features:
- Minimal image size, reducing vulnerabilities and attack surface.
- Support SBOM, provenance attestation, and signed images to ensure supply chain security.
- Support multiple models in a single image.
- Support NVIDIA CUDA and AMD ROCm GPU acceleration.
- Provide contribution guidelines.
For more information and how to fine-tune models or create your own images, please refer to the AIKit website.
