ai-model-gateway

AI Model Gateway

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

Use When

• Designing or implementing the LLM gateway as a control-plane service for a multi-tenant SaaS.
• Replacing direct OpenAI / Anthropic / Bedrock SDK calls in feature code with a routed, audited, cost-attributed path.
• Adding multi-provider fallback to an existing AI feature (Anthropic → Bedrock → OpenAI).
• Enforcing per-tenant token caps, regional routing, and audit at one chokepoint instead of scattered library calls.

Do Not Use When

• The task is the wider AI architecture — start with ai-on-saas-architecture.
• The task is direct provider API exploration / spike — ai-llm-integration is the bare-metal SDK skill.
• The task is the prompt design — use ai-prompt-engineering.

Required Inputs

• The model tier catalogue (Free → distilled; Pro → mid-tier; Enterprise → flagship).
• The provider list, their SLAs, their regions, and contract limits.
• The tenant_ai_binding schema from ai-on-saas-architecture.
• Cost ceiling policy per plan (hard caps vs soft caps).
• Residency commitments per tenant region.

Workflow

1. Read this SKILL.md.
2. Define the internal gateway contract (§1) — HTTP + SDK shape, request envelope, response envelope.
3. Implement the request pipeline (§2): auth → model resolve → rate limit → safety in → provider call → safety out → cost capture → audit → respond.
4. Build the provider adapter layer (§3) so adding a provider is a < 200-LOC change.
5. Add fallback chains (§4) per tier.
6. Add per-tenant token/USD ceilings (§5).
7. Add regional routing (§6) for residency.
8. Wire audit + cost capture (§7) at write time.
9. Implement the kill-switch path (§8).
10. Document SLA and ops (§9).
11. Apply anti-patterns (§10).

Quality Standards

• Adding a new feature requires zero code in the gateway beyond a prompt-id registration.
• Adding a new provider requires < 200 LOC and zero feature-code changes.
• Hot-path overhead added by the gateway < 50ms p95 over raw provider call.
• 100% of AI requests are audited and cost-attributed at request close — never reconciled from invoices.
• Kill-switch flip propagates in < 60 seconds.
• Gateway is the only outbound path; a lint rule rejects PRs that import provider SDKs outside the gateway repo.

Anti-Patterns

• Gateway with feature-specific code paths inside it (gateway becomes a god service).
• No fallback chain — primary provider 429 brings the whole product down.
• Cost computed nightly from logs instead of at request close — tenants get billed days late.
• Rate limit only at the provider layer — one tenant exhausts the global key and noises everyone.
• Audit log writes are best-effort (lost on crash) — compliance and billing diverge.
• Hard caps without a degraded-mode option — tenants hit the cap and the whole product breaks.
• Gateway leaks provider-specific error shapes to feature code — re-coupling.

Outputs

• Gateway HTTP contract + SDK in N languages used by feature teams.
• Provider adapter set with capability matrix.
• Fallback policy per tier.
• Token / USD ceiling policy per plan.
• Region routing table.
• Audit log + cost ledger schema (lives in audit skill).
• Kill-switch UX + back-office wiring (saas-admin-backoffice-tooling).

Evidence Produced

Category	Artifact	Format	Example
Architecture	Gateway contract	OpenAPI doc	docs/ai/gateway-api.yaml
Architecture	Provider capability matrix	Markdown table	docs/ai/provider-matrix.md
Release evidence	Fallback policy per tier	Markdown doc	docs/ai/fallback-policy.md
Operability	Gateway SLO + on-call runbook	Markdown runbook	docs/runbooks/llm-gateway.md

References

• references/llm-gateway-design.md — full design (canonical copy lives in ai-on-saas-architecture/references/).
• references/token-accounting-pipeline.md — how token-in / token-out / cost rolls up.
• Companion: ai-on-saas-architecture, ai-cost-per-tenant-attribution, ai-usage-metering-and-billing, ai-entitlements-and-feature-gating, ai-prompt-injection-and-tenant-safety, ai-observability-and-debugging, saas-rate-limiting-and-quotas.
• Incident primitives: the gateway is the surface that exposes the operator primitives an on-call uses during an incident — kill-switch (feature/agent task), model-pin, prompt-pin, gateway routing pin, per-tenant feature pause, quota cap. Each primitive must propagate in < 60s, log to ai_incident_mitigation_log with (actor, ts, primitive, scope, reason, ticket_id), and be invocable from a back-office UI without writing code or SQL. See ai-incident-response-runbook §3 for the full primitive contract and ai-incident-recovery-and-rollback/references/rollback-patterns.md for the un-pin contract.

§1 Internal Gateway Contract

A single endpoint that feature teams call:

POST /v1/generate
Authorization: Bearer <service-jwt>     # signed by internal auth
X-Tenant-Id: 8421                       # required
X-Feature-Id: support-copilot.answer    # namespaced
X-Trace-Id: trc_01HXY...                # propagate

{
  "prompt_id": "support.answer",
  "prompt_version": "latest",          // or pinned e.g. "v17"
  "variables": { "user_question": "...", "kb_partition_id": "kb_t8421" },
  "retrieval": {                       // optional; gateway can call KB service
    "do_retrieve": true,
    "top_k": 6
  },
  "intent": "answer_question",
  "user_id": 990012,
  "max_tokens_out": 800,
  "temperature": 0.2,
  "stream": false
}

Response:

{
  "request_id": "ai_req_01HXY...",
  "model_used": "Codex-3.7-sonnet",
  "region": "eu-west-1",
  "text": "...",
  "tokens_in": 1840,
  "tokens_out": 412,
  "usd_cost": 0.013824,
  "latency_ms": 1923,
  "fallback_used": false,
  "safety_findings": [],
  "grounding_score": 0.91,
  "citations": [{"chunk_id": "...", "source": "...", "score": 0.83}],
  "eval_sampled": false
}

A streaming variant uses Server-Sent Events. The final SSE event carries the full envelope (cost, latency, model, audit id).

§2 Request Pipeline

1. Authn (service JWT)        — verify signature; resolve calling service
2. Authz (tenant + feature)   — service is allowed to act for this tenant on this feature
3. Resolve binding            — read tenant_ai_binding for tenant
4. Entitlement check          — tenant's plan permits this feature/model
5. Kill-switch check          — ai_enabled = false → 403 fast
6. Rate limit                 — Redis token bucket per tenant per feature
7. Cap check                  — monthly USD/token cap not exceeded
8. Resolve prompt             — prompt registry returns (template, model_hint)
9. Render prompt              — template + variables (sanitised)
10. Safety in                 — prompt-injection classifier on user-supplied variables
11. (optional) Retrieval      — call KB service with tenant_id (no other path)
12. Provider call             — primary; retry once on transient
13. Safety out                — PII scrub, jailbreak detect, grounding check
14. Cost compute              — tokens × price table → usd_cost
15. Audit write               — synchronous; row in ai_requests
16. Cost event                — ai.cost.recorded onto event bus
17. Eval sample               — N% of requests written to eval queue
18. Respond                   — envelope to caller

The pipeline is the gateway. Each stage has a hard timeout; stage failures emit gateway.stage.failed traces.

§3 Provider Adapter Layer

class Provider(Protocol):
    name: str
    models: list[ModelDescriptor]
    regions: list[str]

    async def generate(self, req: NormalizedRequest) -> NormalizedResponse: ...

class AnthropicProvider:
    name = "anthropic"
    models = [
        ModelDescriptor("Codex-3.7-sonnet", ctx=200_000,
                        in_price=3e-6, out_price=15e-6),
        ModelDescriptor("Codex-3-haiku", ctx=200_000,
                        in_price=0.25e-6, out_price=1.25e-6),
    ]
    regions = ["us-east-1", "eu-west-1"]

    async def generate(self, req): ...

Adapter responsibilities:

• Translate normalised request → provider SDK call.
• Translate provider response/error → normalised response/error.
• Surface model capability flags (vision, tools, JSON-mode, streaming).
• Report region routing options.

Anything else (rate limit, retries with backoff, cost compute, audit) lives in the pipeline, not the adapter.

§4 Fallback Chains

Per tier, an ordered list of (provider, model, region) candidates.

tiers:
  enterprise:
    primary:     [anthropic, Codex-3.7-sonnet, region:tenant]
    fallback_1:  [bedrock,   anthropic.Codex-3-5-sonnet, region:tenant]
    fallback_2:  [openai,    gpt-4o, region:tenant_or_us]
  pro:
    primary:     [anthropic, Codex-3.7-sonnet, region:tenant]
    fallback_1:  [anthropic, Codex-3-5-haiku, region:tenant]
  free:
    primary:     [anthropic, Codex-3-haiku, region:any]
    fallback_1:  [openai,    gpt-4o-mini, region:any]

Triggering fallback:

• 5xx, timeout, or RateLimitError after one retry on primary.
• 429 with Retry-After > slo_budget.
• Model deprecation event.
• A safety vote from the in-line classifier (rare).

Record fallback_used=true in the audit row; alert when fallback ratio for a tier exceeds threshold.

§5 Per-Tenant Token / USD Ceilings

The gateway enforces caps at the atomic check-and-increment level using Redis (saas-rate-limiting-and-quotas algorithms). Three ceilings:

• Hard USD cap per month (tenant_ai_binding.monthly_usd_cap). On hit: 429 + quota:ai_usd. Upgrade path link in response.
• Hard token cap per day (rate-shaped). On hit: 429 + quota:ai_tokens_day.
• Soft cap at 80% — fires ai.budget.threshold event for in-product banner and sales-assist email; no enforcement.

For enterprise tenants on a true-up model, replace the hard cap with a paging threshold instead.

§6 Regional Routing

tenant_ai_binding.region drives:

• The provider region called (must match for residency).
• The region of the KB partition called.
• The S3 bucket of the audit payload.

When a region's preferred model is unavailable, the gateway consults a region policy:

• strict: 503 if the region cannot serve. Used for sovereignty.
• degraded: allow cross-region with a region_breach=true flag in audit; emit alert.
• permissive: allow cross-region silently (default for low-sensitivity tenants).

§7 Audit + Cost Capture

Synchronous, in-pipeline, atomic with the response. The gateway returns ONLY after the ai_requests row is committed and the ai.cost.recorded event has been published (or rolled back).

Two writes:

1. Postgres ai_requests row (the legal/compliance record).
2. Redis tenant cost counter increment (the realtime billing view).

A reconciliation job nightly compares Postgres rollups vs Redis to detect drift.

See references/token-accounting-pipeline.md.

§8 Kill-Switch Path

The gateway reads tenant_ai_binding.ai_enabled from a Redis-cached binding (TTL 30s; invalidated by ai.kill_switched event).

On ai_enabled = false:

{
  "error": {
    "code": "AI_DISABLED",
    "message": "AI features are disabled for this tenant. Contact support.",
    "kill_switch_reason": "tenant requested temporary disable"
  }
}

The back-office UI (saas-admin-backoffice-tooling) exposes the toggle plus a feature-scoped variant (tenant_ai_feature_disable table).

§9 SLA & Ops

Metric	Target
Gateway availability	99.95%
Hot-path overhead	< 50ms p95
Audit-write success	100% (any failure = reject request)
Time to roll a new provider	< 1 week from contract to traffic
Time to flip kill-switch	< 60 seconds
Fallback ratio (enterprise tier)	< 1% sustained

Dashboards: gateway QPS, p50/p95/p99 latency by stage, error rate by stage, fallback ratio by tier, cost burn by tenant top-20.

§10 Anti-Patterns

• Building per-feature gateway routes — turns the gateway into a feature factory.
• Streaming responses that don't emit a final envelope event — cost + audit lost on disconnect.
• Async audit writes via best-effort queue — failures cause compliance and billing drift.
• Rate limiting only at the provider — noisy tenants block others.
• Logging full prompts and responses unfiltered — PII exposure; encrypt audit payloads at rest.
• One provider, one model — first outage = total outage.
• Adapter logic leaking into the pipeline (provider-specific if anthropic: ... branches).

§11 Read Next

• ai-on-saas-architecture — broader context.
• ai-cost-per-tenant-attribution — what the gateway feeds.
• ai-usage-metering-and-billing — how the ledger turns into invoices.
• ai-prompt-injection-and-tenant-safety — the safety-in/safety-out logic the pipeline runs.
• ai-observability-and-debugging — traces, replays, debugging.
• saas-rate-limiting-and-quotas — algorithms the gateway uses.