SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsaicr-uat-report

aicr-uat-report

General & other

View repository

Updated 10-16 07:24

About this skill

AICR UAT Report

Reports how the UAT Run workflow (https://github.com/NVIDIA/aicr/actions/workflows/uat-run.yaml) performed over a lookback window, aggregated by service x GPU x intent, with each failure classified as product signal (real test failure) or infra noise. The output feeds the release process: combinations failing on main are the ones to test more closely during RC validation.

When to Use

  • User asks for a UAT report, UAT pass rates, or failing UAT combinations
  • User invokes /aicr-uat-report (optionally with a number of days)
  • Release prep needs the list of service/GPU combos to validate manually
  • User asks why a UAT run failed, or for the debug bundle behind a failure

Do NOT use this skill to re-run, dispatch, or cancel UAT runs.

Inputs

  • days (optional, default 3): lookback window. If the user says "over the last week", pass --days 7. Do not ask — default to 3 when unspecified.
  • Only runs against main (empty aicr_version input) are reported by default; that is the release-process signal. Add --all-versions only if the user explicitly asks to compare against release-tag runs.
  • debug bundles (optional): pass --download-debug <dir> when the user asks why something failed, or when Step 2 classifies a failure as product signal. Skip it for a plain pass-rate report — bundles are tens of MB.

How the Data Works

The workflow's run-name encodes everything needed — no per-job digging: UAT <reservation> <intent> @ <version|main>[ #dispatch_key]. Reservation names are <cloud>-<gpu> rows from infra/uat/reservations.yaml (e.g. aws-h100, gcp-h100, azure-h100, aws-gb200), and cloud maps to service: aws=EKS, gcp=GKE, azure=AKS, kind=Kind (self-hosted nvkind lane). Runs before 2026-07-21 used a title without the intent word; the script derives intent from the nightly dispatch-key cell index (version-outer/intent-inner, training first) and marks those rows "(intent derived)".

Debug bundles

Each per-cloud workflow (uat-aws.yaml, uat-gcp.yaml, uat-azure.yaml, uat-kind.yaml) runs tests/uat/<cloud>/run debug on failure — before teardown, while the cluster is still up — and uploads uat-<cloud>[-<intent>]-debug-<run_id> with 30-day retention. Reusable workflows inherit the caller's run_id, so the artifact hangs off the uat-run.yaml run the report already lists.

The upload is gated on failure() && steps.prep.outcome != 'skipped', so no bundle exists when the cloud job died in bring-up or image build, or when the failure was in a downstream job (evidence ingest) while the cluster job passed. Absence is itself a classification signal, not an error.

What to open, in triage order (if-no-files-found: ignore, so any entry can be missing; cluster-debug/ prefix omitted below):

Open When / what it answers
MANIFEST.yaml Always first — runId, config, resolved recipe + criteria, and failingChecks lifted from report.json
report.json Full validator results; absent if the run died before validate
train-logs/**, serve-logs/** A CUJ check failed (NCCL, inference-perf)
readiness-gate.log Readiness-gate failure — one ===== attempt N section per validate --phase deployment try; the last --- failed validator output (attempt N) --- block names each non-passed validator with its message and stdout, including the Failed resources: list. Runs before #2630 have no such block, only the raw validate output
cr-skyhooks.yaml, node-reboot-fingerprint.txt Tuning-race failure — Skyhook status.status, taints, bootID/kernel
pods-notready.txt, events.txt Scheduling, eviction, OOM
logs-<namespace>.txt The operator owning the failing resource
nodes*, other cr-*.yaml, ns-*.txt Broader node and operator state
snapshot.yaml, recipe.yaml, dry-run.json What was collected / resolved / deployed
evidence-result.json, evidence/pointer.yaml Signed-evidence emit outcome

Procedure

Step 1 — Run the report script

python3 .agents/skills/aicr-uat-report/uat_report.py --days 3

It prints, per version, a Markdown table (Service | GPU | Intent | Pass | Failures) and a "Failure detail" section listing each failing run's timestamp, URL, and the failed job/step names. It is read-only (gh run list / gh run view). If gh is not authenticated, stop and tell the user to run gh auth status.

Step 2 — Classify each failure

Map the failed step name to a failure nature. This drives the RC priority ranking, so classify every failure:

Failed step contains Nature Product signal?
UAT - readiness gate Deployed stack did not converge: a deployment-phase validator kept failing YES — the likely owner is the component behind the failing validator(s), which the script prints as failing validators:
UAT - validate, UAT - prep, CUJ/test phase names Real test failure YES — but the validate step also emits signed evidence, so confirm against report.json (Step 2b) before ranking
Bringup Infra, provision/actuator steps Infra bring-up failure no
Buildx, Build and push, image/GHCR steps CI/image flake no
Validate inputs, UAT - install tagged [apply only: maybe infra] helmfile apply or Argo CD sync failed maybe — recurring = investigate
UAT - install tagged [legacy apply+readiness: ambiguous …] Pre-#2630 run: the step ran both the apply and the readiness gate ambiguous — see below

The script tags install and readiness failures in brackets. A job that has a UAT - readiness gate step uses the split layout, so its install step is apply-only. A job without one predates the split (#2630), and its UAT - install (...) step — including kind's `UAT - install (helmfile apply

  • readiness gate)— covered both halves. For those legacy failures, pull the bundle (Step 2b): areadiness-gate.logwithresult=fail` attempts means the gate failed (product signal); no gate log, or an empty one, means the apply failed (maybe infra). Without a bundle, keep it ambiguous; do not guess.

For readiness failures the failing validators: text comes from the UAT readiness gate failed annotation the phase emits. When it is absent (the annotation failed to post, or the run predates it), read the names from the last failed-validator block in readiness-gate.log.

A retry that went green the same night (same combo, later timestamp, success) downgrades the earlier failure to a flake.

Step 2b — Pull debug bundles (only for product-signal failures)

Skip this step entirely for a routine pass-rate report. Run it when the user asks why something failed, or when Step 2 found a test-phase failure worth root-causing:

python3 .agents/skills/aicr-uat-report/uat_report.py --days 3 \
  --download-debug /tmp/uat-debug --max-downloads 3

Prefer --run <id> (repeatable) over raising --max-downloads: usually only the latest failure per failing combo is worth reading.

Bundles land in <dir>/<service>-<gpu>-<intent>-<run_id>/, each with a printed digest — MANIFEST head, failing checks from report.json, and a one-line contents summary. Read that summary for presence, not file names: a missing evidence/ or report.json says the run died before that stage, which is often the whole diagnosis. For a readiness-gate failure the digest also echoes the last --- failed validator output block from readiness-gate.log; its Failed resources: lines name the resources that never became ready, so start with the logs of the operator that owns them. Then open files per the Debug bundles table above.

Cite file_path:line and the run ID for every finding. Leave the download directory in place — the user may want to keep digging.

Step 3 — Render the report

Produce exactly two artifacts, in this order (see Output Format Reference). Sort the table worst-first: lowest pass ratio at the top; bold the Service/GPU/Intent cells of rows with product-signal failures. Include run URLs as links for at least the most recent failure of each failing combo.

Step 4 — Write the RC validation input

A numbered priority list derived from the table:

  1. Combos with consistent test-phase failures (0/N or repeated validate-phase or readiness-gate failures) — top manual-validation priority. For readiness failures, name the failing validators and the component that owns them.
  2. Combos whose most recent failure is test-phase (even if earlier ones were infra) — deserve a close look.
  3. Combos with infra/CI-only failures — noisy, not product signal; note them but rank low.
  4. Reservations in bring-up (e.g. GB200 while nightly-intents: [], kind lane) — expected churn, call out separately.
  5. Green combos — state them explicitly as lowest priority; a clean bill is information too.

Output Format Reference

## UAT report against `main` (<start>–<end>, N runs)

| Service | GPU | Intent | Pass | Failure nature |
|---|---|---|---|---|
| **AKS** | **H100** | **training** | **0/4** | Real test failures — every run fails at "UAT - validate (all phases)" ([latest](<url>)) |
| EKS | H100 | training | 2/5 | Infra only — 2x bring-up, 1x Buildx CI flake; no test-phase failures |
| GKE | H100 | training | 4/4 | Green |

## RC validation input

1. **<Service>/<GPU>/<intent> is the clear red flag** — <pass ratio,
   failure signature, latest run link, whether it also fails on release
   tags (env issue) or only main (regression candidate)>.
2. ...
N. **<green combos> are solidly green** — lowest manual-testing priority.

Keep failure-nature cells to one sentence; detail beyond that belongs in the RC list, not the table.

Failure Modes

  • gh run list returns nothing — window may predate retention or the workflow was renamed; say so rather than reporting "all green".
  • Unparsed titles in the Notes line — the run-name format in uat-run.yaml changed; read the workflow's current format string and update NEW_TITLE/OLD_TITLE in uat_report.py in the same PR.
  • Unknown reservation (e.g. new cloud) — the script falls back to the uppercased cloud token as the service name; cross-check new rows against infra/uat/reservations.yaml.
  • A combo has very few runs (e.g. 0/1) — flag low sample size instead of declaring it broken.
  • "no debug artifact" — expected for bring-up/Buildx/ingest failures (see Debug bundles). Report it as corroborating the infra classification; do not present it as a tooling problem.
  • "debug artifact expired" — the window exceeds the 30-day retention. Nothing to recover; note it and work from failed step names.
  • cluster-debug/ missing or thin — the collector is best-effort and its cloud credentials can expire on a long failure. Say the bundle is incomplete; do not read it as the cluster being healthy.
  • Readiness gate failed for an infra reason — the gate talks to the API server on every attempt, so expired cloud credentials or an unreachable control plane also fail it. If the failed-validator block is absent or its messages are connection/auth errors rather than resources that are not ready, reclassify to infra.
  • Failing step name disagrees with report.json — trust the bundle. UAT - validate (all phases) + emit signed evidence covers two concerns, so N/N passing checks under a failed step means the evidence leg failed, not the product. Reclassify to infra before ranking it in Step 4.

What This Skill Does NOT Do

  • Does not dispatch, re-run, or cancel workflow runs
  • Does not download raw job logs (gh run view --log); it reports failed step names and, on request, the uploaded cluster debug bundles
  • Does not modify reservations, workflows, or any in-repo file — the only writes are downloaded artifacts under the --download-debug directory