Turns a weekly drift report into a decision. The workflow (https://github.com/NVIDIA/aicr/actions/workflows/registry-drift.yaml) answers "what moved"; this skill answers "what should we take, and what does taking it cost". An AICR component bump is never a version-string edit, which is exactly why the workflow does not open PRs.
When to Use
- The weekly drift digest landed in Slack and someone asks what to do about it
- Release prep needs to know which chart pins can safely advance
- Someone asks whether a specific component should be bumped
Do NOT use this skill to perform a bump. It produces a recommendation; the edit,
the make bom-docs run, and the PR are separate, human-initiated work.
Step 1 — Read the state file
Read drift-state.yaml beside this file before anything else. It records what
was reviewed at which candidate set and why anything was held, so a component
whose candidates have not changed since last week is confirmed, not re-derived.
Update it in Step 5.
Reuse a verdict only after comparing the whole candidate set — current,
latest, and every entry in alternatives[] — against what the entry records.
A verdict keyed on the version pair alone goes stale silently, because a new
candidate can appear while current and latest both stay put: kai-scheduler
sat at v0.16.9 -> v0.20.1 across two runs while a patch v0.16.10 arrived
beside it, and v0.16.10 is the one worth taking. If any candidate was added,
removed, or changed, re-derive.
An entry with no recorded candidates predates that field. Treat it as changed
and re-derive it; do not read the absence as "nothing new".
Step 2 — Resolve the input
In order of preference:
-
An explicit run, downloaded into a scratch directory rather than the working tree (
gh run downloadextracts in place; without--dirthe two report files land as untracked files in this checkout and collide with themselves on a second run):dir=$(mktemp -d "${TMPDIR:-/tmp}/aicr-drift.XXXXXX") gh run download <id> -R NVIDIA/aicr -n drift-report --dir "$dir"Read
drift-report.jsonanddrift-report-raw.jsonfrom"$dir". -
The latest run:
gh run list -R NVIDIA/aicr --workflow=registry-drift.yaml --status=success --event schedule --limit 1 --json databaseIdthen the same scratch-directory download as above (mktemp -d+gh run download <id> -R NVIDIA/aicr -n drift-report --dir "$dir").The default is deliberately the weekly scheduled run on the default branch —
registry-drift.yamlalso runs onworkflow_dispatchand, for a same-repo PR touching the drift surface, onpull_request; every successful path uploads the samedrift-reportartifact name, so an unfiltered "latest successful run" could just as easily be a PR-validation run and put the review against unmerged changes.--event scheduleexcludes both. Reviewing a manual or PR-validation report is still supported — pass--run <id>explicitly (option 1) rather than relying on the default. -
A local
drift-report.jsonpath the user provides -
A pasted Slack digest — it carries counts and a
report artifactlink, no component names, so take the run ID out of that link and download the artifact per option 1. If the artifact has expired, re-derive the drift set fromrecipes/registry.yamland the upstream registry (helm show chartfor HTTP repos,crane lsforoci://). That re-derivation is unfiltered: it does not apply Renovate'sminimumReleaseAge(3 days,.github/renovate.json5) orinternalChecksFilter: "strict", so it can surface a release younger than the cooldown or one Renovate's strict filter would reject. Use it only to identify which components to look at — never to justify a bump on its own; the artifact remains the authoritative source for whether a version is actually eligible.
Confirm schemaVersion is 1. A higher number means this skill is stale —
read tools/drift-report/report.go before trusting the field names.
Report unresolved[] to the user before reviewing anything: those pins are
unknown, not current, and a persistent entry is a broken datasource worth
fixing ahead of any bump.
Read alternatives[] on every row that has one. latest is the largest
step Renovate offers, not the only one and not necessarily the one to take. A
row carrying alternatives has a smaller step available — usually the one worth
recommending, since it crosses fewer upstream changes. Name both in the verdict
and say which you are recommending; a verdict that discusses only latest when
the row offered a minor beside a major has reviewed the wrong upgrade (#2791).
The field is absent when Renovate offered exactly one candidate, which is the
common case. It is additive, so schemaVersion stays 1.
Step 3 — Gather evidence per component
Ordered by how often each is what actually bites.
-
Values-path compatibility. Pull both chart versions and diff their values, then verify every path AICR depends on still exists:
- keys set in
recipes/components/<name>/values.yaml values:blocks inrecipes/overlays/*.yamlandrecipes/mixins/*.yamlthat target the component- the component's
nodeScheduling.nodeSelectorPathsandtolerationPathsinrecipes/registry.yaml
The third is the silent one: a renamed
master.nodeSelectorleaves AICR rendering valid YAML that no longer schedules anything correctly, and no current test catches it. Treat a missing path as a blocking finding. - keys set in
-
Lockstep families. These move together; never recommend a partial bump:
slinky-slurm-operator-crds,slinky-slurm-operator,slinky-slurmmariadb-operator-crds,mariadb-operator,slurm-accounting-mariadbagentgateway-crds,agentgatewayprometheus-operator-crds,kube-prometheus-stack, keyed onappVersion, not chart version (the two chart sequences are unrelated). A lagging CRDs pin also breaksnvsentinelandk8s-ephemeral-storage-metrics, which create resources defined by those CRDs.TestPrometheusOperatorAppVersionLockstepenforces this.prometheus-adapter/prometheus-adapter-ocp,nvidia-dra-driver-gpu/nvidia-dra-driver-gpu-ocp, andk8s-nim-operator/k8s-nim-operator-ocp— these three-ocptwins carry adefaultRepositoryidentical to their base, soBuildReportalready collapses each pair into one report row; name both in the verdict anyway. The other four-ocpcomponents (gpu-operator-ocp,network-operator-ocp,nfd-ocp,cert-manager-ocp) havedefaultRepository: ""and are manifest-only —BuildReportdrops them beforeTracked++, so they never appear in the report at all (they surface inuntrackedComponentsinstead). Do not assume a base component's row already accounts for one of these four; there is no shared row to collapse into.
-
Image and BOM delta. Render old against new and diff the image list.
make bom-docsis the repo's renderer; a new image means new vulnerability surface, and a new registry host meanstools/registry-inventory's allowlist needs extending too. -
CRD and upgrade-record impact. For CRD-bearing charts (kueue, gatekeeper, mariadb, slinky, prometheus-operator-crds, nvsentinel), check whether the transition needs an ADR-021 record: see
pkg/upgradeandtools/check-upgrade-records. -
Upstream breaking changes. Where the chart maps to a GitHub project, read the releases between the two versions (
gh release list -R <owner>/<repo>), looking for removed flags, renamed values, and required migration steps. -
Kubernetes compatibility and blast radius. Compare the new chart's
kubeVersionagainst theK8s.server.versionconstraints of the overlays that pull the component, and check whetherrecipes/checks/<name>/health-check.yamlexists and whether its assertions still hold against the new chart's resource names.
Step 4 — Write the recommendation
Write Markdown to a temp file ("$TMPDIR"/aicr-drift-review-<date>.md) and
summarize it in chat. Rank components by (blocking findings, then update type,
then age of the pin). Per component:
- Verdict: take / hold / defer, one line of why. When the row carries
alternatives, name which version the verdict is about — a bare "take" is ambiguous once more than one is on offer - Evidence: the findings from Step 3 that produced the verdict, with the specific values path, image, or release note that matters
- Cost: what a bump would require — a values-file edit, a new upgrade record, a BOM refresh, a health-check update, an allowlist entry
Do not edit the registry, file issues, or open PRs. Ask before doing anything outward-facing.
Step 5 — Update the state file
Record every component reviewed: the version pair, the full candidate set the verdict was formed against, which candidate the verdict recommends, the date, and for a hold or defer the condition that would change it. Next week's digest repeats the same components by design — this file is what keeps the review from repeating with it.
candidates is what Step 1 compares against, so record it verbatim from the
report rather than summarizing: an entry that omits a candidate makes next
week's run confirm a verdict that never considered it. recommends is only
meaningful when the row had alternatives; omit it otherwise, since the single
candidate is already named by to.
Gotchas
crane lsonghcr.ioworks in the sandbox; NGC (nvcr.io, andhelm.ngc.nvidia.comovercurl) and some TLS paths needdangerouslyDisableSandbox: true. Try sandboxed first.- OCI chart repositories list cosign signature and attestation tags alongside real chart tags. Ignore anything that is not a version.
recipes/overlays/aks.yamlpinskube-prometheus-stackto83.7.0deliberately (#700, declared inversionPinExemptionsinpkg/recipe/version_pin_guard_test.go). Akube-prometheus-stackbump has to reckon with that exemption; do not propose removing it casually.aws-efa's chart is tracked but its device-plugin image is in Renovate'signoreDepsand must be coordinated with EKS add-on releases.