deploy-llms-with-ansible

Purpose

deploy-llms-with-ansible aims to simplify the process of deploying and serving LLMs on virtual machines. It uses Ansible for automated configuration, runs llama.cpp or Ollama via Docker, and restricts access with an IP whitelist, allowing only specified IPs to request the service.

Main Capabilities

• Supports two LLM runtime backends: Ollama and llama.cpp.
• Docker-based containerized deployment, running on Debian-based virtual machines.
• Automated configuration and deployment via Ansible.
• Access control using an IP whitelist; only IPs on the whitelist can request the service.
• The service listens on port 8080 by default and supports requests following the OpenAI API specification.

Usage

Prerequisites

• Ansible is installed on the local machine.
• A Debian-based virtual machine with Docker installed.
• SSH access to the virtual machine is available.

Configuration Steps

1. Depending on the chosen technology (Ollama or llama.cpp), enter the corresponding folder (ollama or llamacpp) and modify the inventory file.
2. Each folder contains an inventory.example.yml template file; copy it to inventory.yml.
3. Modify the configuration items in inventory.yml according to the actual situation.

llama.cpp Configuration Items

• ansible_host: The IP address of the virtual machine.
• ansible_user: The SSH connection username.
• ansible_ssh_private_key_file: The path to the SSH private key file.
• model_url: The URL of the GGUF model file.
• allowed_ip_for_8080: The IP address allowed to access the server (port 8080).

Ollama Configuration Items

• model_name: The model name as defined in the Ollama library (for example, Qwen3-0.6B corresponds to qwen3:0.6b).

Deployment and Requests

• Deploy using Ollama or deploy using llama.cpp (specificcommands are not provided in the material).
• After deployment, the service is available on port 8080, and requests can be made following the OpenAI API specification.
• The model field in the request must exactly match the deployed model:
  • For Ollama: use the same value as model_name in inventory.yml.
  • For llama.cpp: this value does not matter.

Applicable Scenarios

• Scenarios where an LLM needs to be quickly deployed on your own Debian virtual machine and provided as an API service externally.
• Internal or controlled environments where access is restricted via an IP whitelist to enhance security.
• Operations or developers who want to use Ansible to automate the management of the LLM deployment process.

Limitations

• Only Debian-based virtual machines are supported.
• Ansible must be installed locally, and the virtual machine must have Docker installed and support SSH access.
• Access control is based only on an IP whitelist.
• The material does not provide specific deployment commands, the complete content of example requests, or further advanced configuration instructions.