litellm

LiteLLM is an open-source AI gateway (LLM proxy) designed for platform teams, used to unify the entire AI stack behind a single OpenAI-compatible API key. Its core goal is to let everyone in an organization access various models, agents, and MCP, while enabling platform teams to view, control, and optimize every request.

[Purpose and Positioning] As an AI gateway, LiteLLM provides a single point of control in front of numerous LLM providers. It offers applications a unified calling API and gives platform teams centralized cost tracking, budgets, routing, caching, and governance across all models. The material calls it an "AI Gateway for platform teams," open-source and self-hosted.

[Main Capabilities]

1. Unified access: one OpenAI-compatible API covers 140+ providers and 1,800+ models (another place mentions 1,892 models), allowing model changes without modifying application code. It supports SSO single sign-on, permissions by team, project, or application, day-zero support for new models, and access to agents and MCP servers through the same gateway. It can also place self-hosted internal models and fine-tuned models behind the same key.
2. Visibility and control: track usage and spending by key, user, team, organization, tool, agent, and MCP, with enterprise-grade billing allocation. Hard budgets can be set for keys, teams, organizations, and models (reset daily/monthly), and requests stop when the limit is reached; it provides rate limiting and leaked-key protection, model access control and guardrails, and retains audit logs for every request.
3. Cost optimization: change models without changing code; Auto Routing sends simple prompts to cheaper models and difficult prompts to stronger models; it supports load balancing across providers, regions, and keys, as well as lowest-cost routing; it supports response caching and semantic caching (Redis, S3, GCS), and supports prompt compression to reduce token consumption.
4. Deployment convenience: officially provides Helm charts or Terraform modules, official Docker images (including database-bundled and non-root variants), can run on your own Postgres and Redis, autoscales via Kubernetes, supports self-hosting in any environment including fully air-gapped environments, and can also be deployed to AWS, GCP, or Azure with one click.
5. Performance: the material states that LiteLLM Rust AI Gateway adds only 0.66 ms at p99 in benchmarks, with 3.5x lower overhead than the next AI gateway (based on AI Gateway Bench).

[How to Use] It can be deployed via the official Helm chart or Terraform module; use the official Docker image; run on your own Postgres and Redis; supports Kubernetes autoscaling; can be self-hosted in your own cloud or a fully air-gapped environment; and can also be deployed to major cloud platforms with one click. The open-source gateway uses the MIT license and can be self-hosted for free.

[Applicable Scenarios] Suitable for scenarios where platform teams need to uniformly provide model access for the entire organization, centrally manage keys and budgets, perform cost allocation and optimization, route and load balance across multiple providers, and require self-hosting or air-gapped deployment to prevent data leakage.

[Limitations and Notes] The open-source edition includes 140+ LLM provider integrations, Langfuse, Arize Phoenix, LangSmith, and OTEL logging, virtual keys, budgets and teams, load balancing, and RPM/TPM limits, etc. The enterprise edition adds JWT authentication, SSO, RBAC, audit logs, SCIM provisioning, and dedicated support on top of the same open-source core. Specific pricing, billing details, and some integration descriptions are not yet provided in the material.