docker-ai-ml

Docker AI/ML — AI Workloads

Guidance for running AI models and agents on Docker.

When to Use

ALWAYS use this skill when the user mentions:

• "docker ai", "docker model", "model runner"
• "local LLM", "local model", "offline model"
• "docker agent", "AI agent"
• "GPU docker", "nvidia docker"
• "sandbox", "sandbox"
• "LLM deployment", "model deployment"

Docker Model Runner

# List available models
docker model ls

Pull a model

docker model pull ai/gemma3 docker model pull ai/qwen2.5

Run inference

docker model run ai/gemma3 "Explain Docker in one sentence"

Interactive chat

docker model run -i ai/gemma3

API server mode

docker model serve ai/gemma3 --port 11434

→ access at http://localhost:11434/v1

### Install Skills for AI Coding Assistants

```bash
# Install skills for Claude Code
docker model skills --claude

# For Codex CLI
docker model skills --codex

GPU Configuration

# Install NVIDIA Container Toolkit (once)
# Ubuntu/Debian:
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Run with GPU access
docker run --gpus all -it nvidia/cuda:12.6-base nvidia-smi

# Specific GPU
docker run --gpus device=0 -it nvidia/cuda:12.6-base

# GPU + Model Runner
docker model run --gpus all ai/llama3.2

Docker Agent

# agent.yaml — Define a multi-agent system
name: my-coding-agent
model:
  provider: openai
  model: gpt-4o
tools:
  - write_file
  - execute_command
  - web_search
sandbox:
  image: node:22-alpine
  workdir: /workspace

# Run agent with Docker
docker agent run -f agent.yaml "Fix the bug in src/app.js"

Docker Sandbox

# Isolated execution environment for AI agents
sandbox:
  image: python:3.12-slim
  network: none          # No network access
  read_only: true        # Read-only filesystem
  memory: 512m
  cpus: 1

Local LLM Stack

# Ollama (open-source) as alternative
docker run -d --name ollama \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  --gpus all \
  ollama/ollama

# Download and run model
docker exec ollama ollama pull llama3.2
docker exec ollama ollama run llama3.2 "Hello"

Workflow — Recommended Workflow

Step 1: Choose Inference Solution: Docker Model Runner (docker model) preferred, Ollama as backup Step 2: Check Hardware: GPU requires Linux + nvidia-container-toolkit; CPU inference requires 8-32GB RAM Step 3: Pull Model: docker model pull <model> or ollama pull <model> Step 4: Start Service: docker model run or ollama serve + client Step 5: Verify: Send a test prompt to confirm the model responds normally

Gotchas — Common Pitfalls

• GPU not available: Docker Desktop Mac/Windows has limited GPU passthrough. → Recovery: Full GPU support requires Linux host + nvidia-container-toolkit; for Mac, use docker model which can offload inference.
• Model download size: LLMs are large (2-70 GB). → Recovery: docker model pull caches models in ~/.docker/models; check df -h before pulling; use quantized models (Q4/Q8) for smaller footprint.
• Memory limits: LLM inference needs significant RAM (8-32 GB typical). → Recovery: docker run --memory=16g model-runner; check RAM: free -h; use smaller models (7B instead of 70B) if limited.
• Model Runner vs Ollama: Docker Model Runner is Docker-native. Ollama has larger ecosystem. → Recovery: Use docker model for Docker integration; use Ollama for broader model selection; both can coexist.

Boundary — Capability Boundaries (Applicable and Non-Applicable Scenarios)

Category	Scenario	Description
✅ Can Do	Local model inference (Docker Model Runner / Ollama)	Pull and run open-source LLMs without an API key
✅ Can Do	GPU-accelerated inference	Linux host + nvidia-container-toolkit
✅ Can Do	AI Agent building (Docker Agent / Sandboxes)	Multi-agent collaboration, sandbox isolation
⚠️ Conditional	macOS GPU inference	Docker Desktop support is limited; prefer docker model
⚠️ Conditional	Large model (70B+) inference	Requires 32GB+ RAM; consider quantized models
❌ Out of Scope	Training/fine-tuning models	Requires specialized frameworks (PyTorch/TensorFlow), out of scope for this skill
❌ Out of Scope	Cloud API calls (OpenAI/Claude)	Not local inference; use the corresponding SDK
❌ Out of Scope	Production model serving (K8s)	Use docker-production + K8s deployment

When NOT to Use This Skill

❌ Skip	✅ Use Instead
Docker basics	docker-basics
Building custom model images	docker-build
Production model serving (K8s)	docker-production
Cloud AI APIs (no Docker needed)	Direct API access

Security & Stability

• Run model containers with --read-only when possible — models are read-only weights.
• Limit network access for model containers. Local inference doesn't need internet.
• GPU passthrough gives the container direct hardware access. Combine with --security-opt for isolation.
• Monitor GPU memory usage: nvidia-smi for utilization.

📚 Official Documentation References

Documentation	Address
Docker AI Overview	https://docs.docker.com/ai-overview/
Docker Model Runner	https://docs.docker.com/ai/model-runner/
Docker Model Runner Getting Started	https://docs.docker.com/ai/model-runner/get-started/
Docker Agent	https://docs.docker.com/ai/docker-agent/
Docker Sandboxes	https://docs.docker.com/ai/sandboxes/
AI/ML Examples	https://docs.docker.com/reference/samples/ai-ml/

🧭 Docker Skills Journey

📍 You are here: docker-ai-ml — AI/ML Workloads

← Prev: docker-testcontainers — Integration testing

FAQ

Q1: How do I get started with this skill quickly? A: Refer to the Quick Start section above and follow the steps.

Q2: What should I do if I encounter version incompatibility issues? A: Check dependency versions, use lock files to pin them, and refer to the Common Pitfalls section.

Q3: How do I use this in a production environment? A: Refer to the Best Practices section, ensure the configuration is correct, and set up monitoring and logging.

Q4: How can performance be optimized? A: Refer to the performance optimization documentation and use techniques such as caching and indexing.

Q5: How can I contribute or report issues? A: Submit an Issue or Pull Request in the GitHub repository.

Q6: Is Chinese supported? A: Chinese documentation and Chinese comments are supported; see the domestic adaptation section for details.