ollama-api

Ollama API

Ollama exposes a local HTTP interface for LLMs. Prefer /api/chat for multi-turn conversations; use the OpenAI-compatible endpoint when you want to reuse existing clients.

Stack and scope

Endpoint	Role
POST /api/chat	Chat messages + streaming
POST /api/generate	Single prompt completion
POST /api/embeddings	Vectors
GET /api/tags	Pulled models
POST /v1/chat/completions	OpenAI-compatible

Default base: http://localhost:11434. Model must exist locally (ollama pull).

Non-negotiable rules

• Run and configure Ollama on the server/backend — do not expose the raw port to the internet without auth/proxy.
• Handle timeouts and cancellation; generation can be long.
• Streaming: consume NDJSON (stream: true) until done.
• Pin the model name via config/env; fail clearly if the model is not pulled.
• Do not send user secrets in prompt logs.

Chat

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "system", "content": "You are concise."},
    {"role": "user", "content": "Explain iterators in one sentence."}
  ],
  "stream": false
}'

const res = await fetch("http://localhost:11434/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: process.env.OLLAMA_MODEL,
    messages,
    stream: false,
    options: { temperature: 0.2 },
  }),
});
const data = await res.json();
const text = data.message?.content;

• Roles: system / user / assistant / tool according to model support.
• options (temperature, num_ctx, etc.) per request or modelfile default.
• Tools/function calling: use the field tools + messages tool when the model supports it.

Streaming

• stream: true → JSON lines (NDJSON); last with "done": true.
• On the app server: proxy the stream to the client (SSE) if the browser does not talk to Ollama directly.
• Propagate abort (AbortSignal / reader cancel) up to the upstream.

OpenAI-compatible

// baseURL: http://localhost:11434/v1
// apiKey: "ollama" (placeholder)

• Useful for libs that already speak Chat Completions.
• Not every OpenAI parameter is honored — validate on the target model.
• Embeddings: /api/embeddings or your client's compatible route. Index + retrieval: rag.

Ops and quality

• ollama pull <model> in provisioning/CI; pin tag/digest when possible.
• GPU/CPU and num_parallel affect latency — measure p95 on real hardware.
• Concurrency: queue in the app if Ollama saturates.
• Health: GET /api/tags or short generate on readiness.

Anti-patterns

• Browser → Ollama without proxy/auth on an untrusted network
• Assuming every model supports tools/vision
• Blocking the event loop without a timeout while reading the stream
• Logging full prompts with PII
• Swapping models in prod without quality/latency testing

Completion criteria

• Configurable model and present on the host
• Chat or generate with errors/timeouts handled
• Streaming cancellable if the UX is stream
• Ollama not publicly exposed without protection
• Stable message contract for the app