litellm-routing-reliability

LiteLLM Routing Reliability

Workflow

1. Define user-facing model groups, deployments, regions, capacity, and failure domains.
2. Verify the currently supported routing strategy, retry, timeout, and fallback schema.
3. Configure bounded retries and an overall timeout, then define an ordered fallback.
4. Evaluate Redis when multiple instances share routing or health state.
5. Test normal routing, 429, timeout, primary failure, and all-deployments failure separately.
6. Use readiness as the criterion for receiving traffic and liveness as the criterion for restarting; do not treat something that actually calls the model as a /health high-frequency free probe.
7. Report the deployment actually hit, the number of retries, latency, and cost impact.

When configuration examples, fallback mocks, or health endpoint details are needed, read Experiments and References。

Security Baseline

• Fallback must not bypass model permissions, data regions, guardrails, or budget limits.
• Do not use the same provider or same region as the sole disaster failover.
• Do not configure unlimited retries or unbounded timeouts.
• Tests must not generate 429s by using large volumes of paid traffic.

Acceptance

• The order of primary, retry, and fallback can be proven by logs or traces.
• mock_testing_fallbacks is explicitly placed in the request body or SDK call parameters.
• Return a controlled error when all providers fail.
• The purpose, frequency, authentication, and cost of health probes are all defined.