add-model

Add Model

Add a new LLM to a live LiteLLM proxy.

Setup

Ask for these if not already known:

LITELLM_BASE_URL  — e.g. https://my-proxy.example.com
LITELLM_API_KEY   — proxy admin key

API reference: https://litellm.vercel.app/docs/proxy/model_management

Ask the user

1. Public model name — what callers will send in "model": "..." (e.g. gpt-4o, my-claude, llama3)
2. Provider — pick from the table below
3. Credentials — whatever that provider needs

Provider table

Provider	litellm_params.model	Extra params
OpenAI	openai/gpt-4o	api_key
Azure OpenAI	azure/<deployment-name>	api_key, api_base, api_version
Anthropic	anthropic/claude-3-5-sonnet-20241022	api_key
AWS Bedrock	bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0	AWS creds via env
Google Vertex	vertex_ai/gemini-1.5-pro	vertex_project, vertex_location
Ollama	ollama/llama3	api_base (e.g. http://localhost:11434)
Groq	groq/llama-3.3-70b-versatile	api_key
Together AI	together_ai/meta-llama/Llama-3-70b	api_key
Mistral	mistral/mistral-large-latest	api_key

Full list: https://docs.litellm.ai/docs/providers

Run

curl -s -X POST "$BASE/model/new" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "<public-name>",
    "litellm_params": {
      "model": "<provider/deployment>",
      "api_key": "<key>",
      "api_base": "<base_if_needed>",
      "api_version": "<version_if_azure>"
    }
  }'

Test it

After adding, verify it routes:

curl -s -X POST "$BASE/chat/completions" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<public-name>",
    "messages": [{"role": "user", "content": "say hi"}],
    "max_tokens": 10
  }'

Output

Show model_id from the response — needed to update or delete the model later. Report pass/fail from the test call.