SuperAPI AI Directory

provider-utilization-scorecard

General & other

View repository

Updated 10-16 07:16

About this skill

Provider Utilization Scorecard

Use this when the goal is to operationalize weekly credit utilization across Codex, Codex, and Gemini instead of just giving static advice.

Canonical inputs

  • Quota latest snapshot: config/ai-tools/agent-quota-latest.json
  • Quota history log: ~/.agent-usage/weekly-log.jsonl
  • Provider activity logs: logs/orchestrator/{Codex,codex,hermes,gemini}/session_*.jsonl
  • Provider session audit: analysis/provider-session-ecosystem-audit.json
  • Human-readable report: docs/reports/provider-utilization-weekly.md
  • Machine-refresh wrapper log: logs/quality/provider-utilization-refresh-*.log

Canonical commands

Refresh quota + utilization artifacts:

bash scripts/cron/provider-utilization-refresh.sh

Show dashboard:

uv run --no-project python scripts/ai/credit-utilization-tracker.py --dashboard

Reinstall cron if schedule changed:

bash scripts/cron/setup-cron.sh --replace
crontab -l | grep provider-utilization-refresh

Validate tests/schedule:

uv run pytest tests/analysis/test_credit_utilization_tracker.py tests/cron/test_provider_utilization_refresh.py
uv run --no-project python scripts/cron/validate-schedule.py

Core approach

  1. Refresh quota snapshots first using scripts/ai/assessment/query-quota.sh --refresh --log.
  2. Build the weekly scorecard from both quota snapshots and session-activity exports.
  3. Prefer quota-based utilization only when the provider exposes real weekly data.
  4. Fall back to activity-vs-recent-peak when quota telemetry is missing or only estimated.
  5. Schedule the refresh every 4 hours so weekly utilization is actionable, not stale.

Interpretation rules

  • utilization_basis=quota is strongest; use it for routing decisions.
  • utilization_basis=activity_vs_recent_peak is directional only; use it to spot likely underuse, not exact headroom.
  • Hermes is an orchestrator signal, not a paid-provider utilization target.
  • Current underutilization alerts should focus on Codex/Codex/Gemini, not Hermes.

Known provider realities

Codex

  • Real weekly quota may be unavailable in agent-quota-latest.json depending on the local source.
  • Do not compute fake usage from pct_remaining if the snapshot is essentially unavailable.
  • If Codex quota is unavailable, report activity fallback explicitly.

Codex

  • week_messages and weekly_limit from history.jsonl are strong enough for real quota-based utilization.
  • Codex is usually the cleanest signal for bounded implementation/test workloads.

Gemini

  • Current telemetry may only be today_messages / daily_limit with source=estimated.
  • Treat Gemini quota numbers as weak; prefer activity fallback and mark the limitation clearly.

Critical implementation pitfall

When exported session logs lack runtime session_id, DO NOT fall back to a per-record key like provider:file:tool:ts for session counts. That massively overcounts sessions. Instead, fall back to the session_YYYYMMDD.jsonl file identity so session counts remain sane.

Another pitfall

Do not treat missing numeric quota fields as zero during snapshot merging. If week_messages is absent, keep it absent; otherwise the tracker can incorrectly infer weekly_limit usage and produce bogus 100% utilization.

Outputs to maintain

  • config/ai-tools/provider-utilization-weekly.json
  • docs/reports/provider-utilization-weekly.md
  • scheduled task provider-utilization-refresh in config/scheduled-tasks/schedule-tasks.yaml
  • wrapper scripts/cron/provider-utilization-refresh.sh
  • tests:
    • tests/analysis/test_credit_utilization_tracker.py
    • tests/cron/test_provider_utilization_refresh.py

Recommended follow-on work

After the scorecard exists, the next high-value layer is routing automation:

  • if Codex utilization is low, surface bounded implementation/test/refactor work
  • if Gemini utilization is low, surface research/recon/risk-scan batches
  • if Codex utilization is low and quota is trustworthy, route adversarial review and long-context synthesis there

Weekly credit-burn operating model

When the user asks to maximize provider credits across Codex/Codex/Gemini, convert the scorecard into an executable package pipeline, not just a narrative report.

Target burn curve for providers with real weekly quotas:

  • Day 1: 18% cumulative
  • Day 2: 36% cumulative
  • Day 3: 54% cumulative
  • Day 4: 72% cumulative
  • Day 5: 90% cumulative
  • Days 6-7: preserve the final 10% for failures, review, CI repair, and closeout

For providers with daily caps instead of true weekly front-loadable quotas, do not claim 90% weekly by day 5 if the math is impossible. Example: Gemini at 1000/day can only reach 5/7 = 71.4% of a seven-day total by day 5. Manage those as daily burn targets instead: use roughly 80-90% of each day’s capacity with a small reserve.

Always-ready package queues

To keep feature/issue work continuously executable as long as AI credits remain, maintain four queues:

  1. Recon-ready Gemini packages

    • target inventory: 10-15 packages
    • package size: 5-6 related research/recon/source tasks per session
    • focus: raw data, standards/source discovery, competitor/GTM scans, wiki gap discovery, issue expansion
  2. Plan-review packages

    • target inventory: 10+ issues in or near plan-review
    • focus: feeding the approval pipeline before implementation capacity runs dry
  3. Plan-approved Codex implementation packages

    • target inventory: 8-12 approved packages
    • focus: bounded implementation, tests, calculation modules, parametric outputs, CI/harness repair, static-site/GTM slices
    • if Codex utilization is low, this queue is usually the bottleneck, not Codex capacity
  4. Implementation-review / closeout packages

    • target inventory: 5-8 packages
    • focus: adversarial implementation review, CI evidence, closeout comments, future-issue extraction

Value-chain routing model

For the recurring ACE/workspace-hub pipeline, route work by value-chain stage:

  • raw data -> llm-wiki: Gemini for source discovery and gap scans; Codex for contracts/review; Codex for promotion pipeline/checkers/tests
  • llm-wiki -> calculation code: Codex for semantic contracts and plan review; Codex for TDD implementation; Gemini for standards/source cross-checks
  • calculation code -> parametric outputs: Codex for generators, reports, dashboards, and tests; Codex for acceptance review; Gemini for benchmark/reference scans
  • parametric outputs -> website/GTM: Gemini for prospect/company research; Codex for page/data/CTA implementation; Codex for narrative synthesis and final GTM review
  • control-plane enablers: Codex for review-runner/harness fixes, Codex for governance/review, Gemini for audit/recon only

Package lifecycle gate

Every provider package should follow: GitHub issue -> resource intelligence -> canonical plan -> adversarial plan review -> user approval -> status:plan-approved -> implementation -> adversarial implementation review -> closeout.

Never dispatch implementation from status:plan-review. A package is execution-ready only when it has a GitHub issue, canonical plan under docs/plans, plan review artifacts, explicit approval, status:plan-approved, an agent/provider label, and clear closeout criteria.

Dispatch rule

Use this loop after refreshing the scorecard:

  • If Codex is below the burn line, dispatch the next status:plan-approved implementation/test/refactor package.
  • If Gemini daily use is below target, dispatch the next 5-6 task recon/research batch.
  • If Codex has review backlog, dispatch plan or implementation review packages.
  • If no approved implementation work exists, pause coding and spend Codex/Gemini on refilling the plan-review and approval pipeline.
  • If a provider is ahead of target, reserve it for reviews, failures, and closeout.