aisix

Purpose

AISIX is an open-source, Rust-native AI gateway for establishing a unified control point in front of LLM and AI agent traffic. It unifies all model providers under a single OpenAI-compatible API, letting platform teams handle routing, governance, security hardening, caching, and observability management in one place. The gateway can run within the user's own infrastructure, with real-time AI traffic calling each provider directly, without passing through any control plane.

AISIX was built by the original creators of Apache APISIX, and this repository, AISIX OSS, is the open-source product.

Key Capabilities

• Unified API entry point: Provides an OpenAI-compatible proxy (default port :3000), covering chat/completions, completions, responses, embeddings, rerank, images/{generations,edits}, audio/{speech,transcriptions,translations}, videos (submit→poll→fetch), files, batches, fine_tuning/jobs, realtime, GET /v1/models, as well as the root-level /passthrough/:provider/* escape hatch. Natively supports SSE streaming, tool/function calling, JSON mode, vision/multimodal input, and reasoning content.
• Anthropic Messages API: POST /v1/messages as a first-class route, which can act on any configured upstream; when the model points to a non-Anthropic provider, requests and responses (including streaming) are converted bidirectionally.
• Routing and failover: Virtual/routed models support six strategies—round_robin (smooth weighted round-robin), consistent_hash (session affinity by header / cookie / API key / client IP), failover, and metric-based least_cost, least_latency, least_busy. It also supports layering by target priority (active/backup pools), retry budgets, cooldowns, tag-conditional targets, and per-attempt timeouts.
• Ensemble models (Ensemble): Fan out a single request concurrently to a group of models, then have a judge model synthesize a single answer, with a configurable minimum successful-response threshold.
• Semantic routing: A virtual model dispatches by request meaning—embeds the prompt, scores it against each route's example statements, and routes to the best match or the default route.
• Rate limiting and concurrency: RPS/RPM/RPH/RPD + TPM/TPD + concurrency caps, combinable with AND across caller key, model, and policy scopes (api_key / model / team / member / team_member). Counters are per-process by default; a Redis backend can be used to share across replicas.
• Guardrails: Enforce content policies on input and output, either in-process or via a provider—keyword/regex, built-in PII detection and redaction, Presidio, Lakera, OpenAI Moderation, AWS Bedrock Guardrails, Azure AI Content Safety (Prompt Shield + text moderation), and two Alibaba Cloud services. Blocking returns 422 content_filter; monitoring mode records what "would have happened" but does not block.
• Caching: Exact-match response caching, supporting per-policy TTL and model/key scope matchers; in-memory and Redis backends; cost-savings telemetry on every hit. In addition, automatic prompt caching can be enabled for direct Anthropic models to inject cache breakpoints, letting callers obtain provider-side prompt discounts without changing the request.
• MCP gateway: Fronts registered upstream MCP servers in front of /mcp, with credentials held by the gateway, supporting per-server tool namespacing and per-caller access. It serves all Streamable HTTP revisions from 2025-03-26 to stateless 2026-07-28, with no downstream session required. Upstream uses initialize by default, or server/discover with protocol_version: "2026-07-28". CI runs the applicable tool-only protocol scenarios from the official MCP suite. It can also expose OpenAPI descriptions as REST APIs in the form of MCP tools.
• A2A agent gateway: Fronts in front of /a2a/:agent A2A (Agent-to-Agent) agents, exposing each agent's card over JSON-RPC 2.0 and rewriting URLs to the gateway address.
• Inbound authentication: caller API keys (SHA-256 hashing, model allowlists, expiry, rotation), or OIDC/JWT bearer tokens validated against registered providers (Entra ID, Okta, Google Workspace, or any OIDC issuer), with JWKS caching.
• Observability: Prometheus /metrics, structured per-request access logs, usage events, OTLP/GenAI span export (Langfuse, Honeycomb, Grafana Cloud, or any OTLP receiver), plus dedicated Datadog and Alibaba Cloud SLS log exporters and object storage (S3/GCS/Azure Blob) telemetry.
• Declarative configuration: a single resources.yaml carries all ten resource collection types (provider keys, models, caller keys, guardrails, MCP servers, A2A agents, cache policies, observability exporters, rate limit policies, OIDC providers), and is validated using the same JSON Schema as the runtime. aisix validate can check the file offline; SIGHUP performs an atomic reload.
• Operational endpoints: /livez and /readyz on the proxy listener; /status/config, /status/ready, /status/models, and Prometheus /metrics on the dedicated metrics listener (:9090). The admin listener (:3001) additionally provides a read-only resource UI, OpenAPI 3 with Scalar UI, and a playground. Resources are managed declaratively—via resources_file (SIGHUP reload) or by writing directly to etcd—rather than through the admin listener; its legacy write endpoints have been removed.

Usage

The official quick-start path is: run it with a single container, with no control plane, database, or configuration store—the gateway reads all dynamic resources from a single declarative resources.yaml; then call the gateway just like you would call OpenAI. Edit resources.yaml and send SIGHUP (docker kill -s HUP aisix) applies changes without a restart; an invalid file is rejected as a whole, and the last valid configuration continues to serve. Before startup, you can validate the file with aisix validate --resources resources.yaml. For a multi-replica cluster, point to etcd instead (resources_file and etcd are mutually exclusive).

Local build prerequisite: the Rust toolchain pinned in rust-toolchain.toml. Docker is only needed when testing etcd, Redis, or provider simulators.

Use cases

Suitable for platform teams that need to uniformly proxy traffic from multiple LLMs and AI agents within their own infrastructure, for multi-provider routing and failover, model invocation governance, rate and concurrency control, content guardrails, response caching, MCP and A2A traffic fronting, inbound authentication, and usage observability. The gateway runs in the user's environment on the data plane and calls providers directly.

Product form and limitations

The open-source edition AISIX OSS (this repository) can run via declarative configuration or etcd without a control plane. The same gateway binary also serves as the data plane for the commercial products AISIX Enterprise and AISIX Cloud: the former is hosted by the customer's control plane, and the latter is hosted by API7. In all three products, the gateway runs in the user's environment and calls providers directly; real-time AI traffic does not pass through any control plane, and the proxy API is completely identical. Budgets and spending caps are features of AISIX Enterprise and AISIX Cloud—the gateway executes the control plane's decisions. The open-source edition is licensed under Apache-2.0 and can run permanently free of charge in your own infrastructure.

Items on the roadmap (tracked in issues) include: semantic (embedding similarity) response caching, more observability sinks (Langsmith, Helicone, Slack alerts), prompt templates managed as gateway resources, and a Llama-Guard guardrail provider. Delivered since that list was last written: MCP gateway, A2A agent gateway, OIDC/JWT inbound authentication, Redis-backed distributed rate limiting, and Lakera, Presidio, PII, and OpenAI Moderation guardrails.

On the supported provider side, AISIX distributes through five nativeadapter families—they are independent wire-protocol bridges, not a single generic relabeling; regardless of the upstream protocol, the client-facing API always remains OpenAI-shaped. It also includes dedicated handling for vendor-specific behavior (such as DeepSeek reasoning content) and dedicated rerank / embeddings vendors (Cohere, Jina).

Regarding specific fees and pricing, the materials do not currently provide them.