openrouter-provider-ranking

OpenRouter Provider Ranking

Rank the endpoint providers of a single OpenRouter model slug for a specific workload. Never call the order "globally optimal": it is optimal only against the stated constraints, weights, token profile and available telemetry.

Mandatory rules

1. Apply hard constraints first, score second. Never let high TPS or a low price compensate for an incompatibility.
2. For tool calling default to native-exacto: the :exacto model suffix, no provider.sort, no provider.order. Your own score in that mode is diagnostic, not a replacement for OpenRouter's private telemetry.
3. Use manual only when the user requires a deterministic order, when your own production telemetry outweighs Exacto, or when a failover chain has to be pinned explicitly.
4. Never combine :exacto with provider.sort — an explicit sort wins. Never add provider.order in native-exacto mode.
5. Price on the real token profile, not on headline input/output rates. Include cache read/write, per-request fees, conditional pricing overrides and the observed cache hit rate.
6. Do not invent missing Exacto, benchmark, cache or tool-success metrics. Apply a conservative prior and an uncertainty penalty, and name the gaps explicitly.
7. For multi-turn workloads pass a stable session_id. Remember that a manual provider.order disables OpenRouter's sticky provider routing, and session_id does not restore it.
8. Never store the API key in the skill, a config, a log or the output JSON. Read it only from OPENROUTER_API_KEY or the environment variable the user names.
9. A catalogue field is a hypothesis; a measurement is a fact. Do not hand back a ranking as the answer: always propose the minimal verification of the top candidates against the constraints that were asked for (step 5), and say plainly whether it was run.
10. The comparison row is endpoint × reasoning effort, not the endpoint. Changing effort moves price, latency and quality more than changing provider does — and "none" means "omit the field", not "no reasoning".
11. Compute price from components on the token profile (in, out, cache_read, cache_write, per-request fee). Headline and effective price swap places: the most expensive rate card produced the lowest cost per step at a 99.996 % cache hit.
12. Measure cache, latency and reliability on the profile that will ship. Cache hit on a repeated identical prompt (99 %) and in an agent run (68 %) are different numbers; TTFT on a short prompt is 4-6x optimistic; and fitness for a single request does not predict behaviour under sequential load.

Inputs

Collect or estimate:

• the model slug, e.g. deepseek/deepseek-v4-flash-0731;
• uses_tools, streaming, required parameters, context and output limits;
• expected prompt/completion tokens and requests per session;
• cacheable prompt fraction, token-level cache read/write rate, response-cache hit rate;
• hard caps: price, latency, TPS, uptime, quantization, moderation, ZDR/data policy;
• the goal: quality, balanced, interactive latency, cost or batch throughput;
• your own telemetry by provider tag, if any exists.

When the input is incomplete, use the agentic-balanced profile. For tool calling set uses_tools=true. Do not assume a non-zero cache hit rate without observations or a defensible workload model.

Procedure

1. Classify the workload

Pick one profile:

• agentic-balanced — the default for agents and B2B SaaS;
• agentic-quality — tool correctness and reliability outweigh price;
• interactive — minimise TTFT and end-to-end latency;
• cost — minimise expected cost under an SLO;
• batch — throughput and cost for long completions.

For a goal outside these, override weights; the sum after normalisation must be positive. The formula: references/scoring.md.

2. Fetch fresh endpoint metrics

Preferred path — the OpenRouter Endpoints API through the bundled script:

python3 scripts/rank_providers.py \
  --model deepseek/deepseek-v4-flash-0731 \
  --config assets/config.example.json \
  --format markdown \
  --output recommendation.md

For a reproducible or offline analysis:

python3 scripts/rank_providers.py \
  --endpoints-file endpoints.json \
  --config config.json \
  --observations telemetry.jsonl \
  --previous-ranking previous-result.json \
  --output result.json

The Endpoints API gives the provider tag, pricing, quantization, context and output limits, supported parameters, uptime, and latency and throughput percentiles. When reading a performance page, carry any provider-specific Auto Exacto or benchmark values into observations by exact tag; never guess the match from a display name when a tag exists.

Input formats: references/input-formats.md.

Read the metrics as they are defined, not as they read. latency_* arrives in milliseconds — 1624.8 is 1.6 s. A p90 throughput is the fast tail and a p90 latency is the slow one, so rank on p50 and report both. The storefront's own numbers are collected on short prompts; see step 5 and references/storefront-traps.md.

3. Add production telemetry

Quality signals, in order of priority:

1. your own tool-call/schema success for the same model, prompt class and provider tag;
2. provider-specific Exacto or benchmark values from the performance page;
3. the endpoint performance/uptime API;
4. a conservative prior when nothing is available.

Collect at least: provider_name or provider_tag, success, tool success, prompt/completion/cached/cache-write tokens, TTFT, generation time or TPS, and total cost. For rates from a small sample use a Wilson lower bound, not the raw percentage.

4. Run the ranking and check the result

The script should:

• drop inactive and incompatible endpoints;
• resolve pricing overrides as of the request;
• compute the expected cost for the workload;
• blend OpenRouter percentiles with your own observations by sample confidence;
• normalise cost, TPS, TTFT and E2E against the current eligible pool;
• apply the quality, reliability, cache, fidelity and uncertainty components;
• stabilise the order against a previous result;
• pick a fallback chain with provider-family diversity when the score gap allows.

Check the exit code. On 4 do not relax anything silently: show which constraints eliminated every endpoint and propose the smallest relaxation.

5. Verify the candidates with real requests — mandatory

A ranking is built from catalogue fields, and several of them are wrong often enough to change the answer: supports_implicit_caching reads false on an endpoint that caches 99 % of the prefix; storefront TTFT is taken on short prompts and is 4-6x optimistic; and the cheapest endpoint on a bench may serve no request at all under sequential load. The full list, each with its reproduction: references/storefront-traps.md.

So never hand back a ranking as the final answer. Always propose the minimal verification of the top candidates against the same hard constraints the score used, and run it where you can:

python3 scripts/probe_endpoints.py \
  --model deepseek/deepseek-v4-flash-0731 \
  --providers coreweave/fp8,fireworks,siliconflow/fp8 \
  --prompt-file real-prompt.txt \
  --max-tokens 600 --runs 3 \
  --min-tps 50 --max-ttft 6 --require-cache --require-max-tokens

The script sends --runs identical requests per endpoint and reports the measured TTFT, TPS, cost, cache hit share, completion tokens and error class, then judges each against the thresholds you pass. Exit code 4 means none survived — the ranker's own convention; do not relax the thresholds silently.

What must be passed:

• --prompt-file with your own prompt once a decision is actually being made. The --prompt-tokens filler is fine for a rough cut, but TTFT depends on size and contract compliance depends on content.
• --runs >= 2, but do not read a cache miss from filler as a verdict: a fresh prefix does not become cacheable inside a short probe. The same endpoint gave 99 % over two calls on a prompt that was already in use and 0 % over four on a new one. A hit proves caching; a miss proves it only with --prompt-file.
• Thresholds taken from the hard constraints, not invented: the check must answer the user's question, not a generic one.

What the check does not do: it does not measure exact throughput and it does not certify fitness. A failure is a conclusion; a pass is "on this profile, in this window, over N calls". For an agentic workload add a run of sequential calls — fitness for a single request does not predict it.

A separate one-request check of the data policy, whenever the constraints carry data_collection: deny or zdr: max_tokens: 1, allow_fallbacks: false, one request per candidate. Distinguish a policy refusal (404 No endpoints found matching your data policy) from a provider 429 — different causes, different conclusions.

6. Choose the routing mode

Condition	Mode	What to send
Tool calls, no fixed order required	native-exacto	model: <slug>:exacto, filters/preferences, no sort/order
A pinned failover chain, or strong first-party telemetry	manual	provider.order from the ranking
One simple goal with no custom score	native OpenRouter	provider.sort: price/throughput/latency; the bundled ranker is optional

OpenRouter-specific interactions and limits: references/openrouter-routing.md.

Response format

Always return:

1. the chosen mode and a short justification;
2. the eligible ranking table: provider tag, score, expected cost per request, TPS percentile, TTFT, E2E, uptime, quality confidence and cache hit rate;
3. a ready JSON request fragment;
4. the excluded providers, each with its reason;
5. coverage and warnings, and the list of missing signals;
6. the minimal verification plan — a ready probe_endpoints.py command with thresholds taken from the hard constraints, and its result if it was run. Call the ranking a hypothesis until that check has been made;
7. the refresh rule: recompute after noticeable drift, a price or endpoint change, or enough new telemetry. Never pin an order indefinitely.

In native-exacto, keep the diagnostic ranking and the authoritative runtime ordering by Exacto clearly apart.

Validation checklist

Before returning a result, check that:

• provider tags come from a fresh API call or file, not from memory;
• hard constraints were applied before the score;
• prices were converted from USD/token to readable USD/M and to workload cost without double conversion;
• p90 throughput is read as the fast tail — the value about 10 % of requests exceed — and p90 latency as the slow one. Rank on p50; a provider whose p90 is 57 and p50 is 33 is not the fastest, and reading p90 as a floor already put one first;
• latency_* is read as milliseconds, while provider.max_price is passed in USD per million tokens and endpoint pricing is stored in USD per token;
• provider.order is absent in native-exacto;
• service-tier tags (/fast, /flex) were not included by accident;
• cache-heavy sessions carry a stable session_id, and the consequences of a manual order are noted;
• a small sample is not over-read;
• the final request JSON is syntactically valid;
• the minimal verification of the top candidates was proposed — and run where possible — and a ranking without it was called a hypothesis;
• supports_implicit_caching, TTFT and throughput were not taken on faith wherever the conclusion depends on them.

Bundled resources

• scripts/rank_providers.py — the standalone ranker and routing-fragment generator.
• scripts/probe_endpoints.py — the minimal verification of candidates with real requests: TTFT, TPS, cost, cache hit, max_tokens compliance and error class; judges each against the thresholds passed, exit code 4 when none survives.
• scripts/validate_skill.py — self-check of frontmatter, resources, syntax and unit tests.
• assets/config.example.json — a config for a tool-using agent workload.
• assets/observations.example.json — aggregated provider observations.
• assets/telemetry.example.jsonl — raw request-level observations.
• tests/trigger-evals.json — positive and negative activation queries for the description.
• references/scoring.md — the formula, normalisation and profiles.
• references/input-formats.md — the config and telemetry schema.
• references/openrouter-routing.md — semantics OpenRouter routing/caching.
• references/storefront-traps.md — reproduced disagreements between catalogue fields and endpoint behaviour; read it before ranking.

To validate the package, run:

python3 scripts/validate_skill.py