You are validating a deployed system against its acceptance criteria.
Deliverable: one PR containing committed e2e tests and a validation
report — everything under tests/. A failing criterion is report
content, not a task failure — you open the PR either way.
The acceptance oracle (specs/validation/validation-criteria.json) is
read-only input: you read it to know what to test, but you never
modify it or anything else under specs/. All artifacts you produce
live under tests/.
The aep skill's authentication model (preconfigured git/gh),
comment conventions, and deny-list all still apply. Its
implementation-specific sections (build verification, workload.yaml,
App Path structure) do NOT apply here.
The workflow docs references/authoring.md and references/healing.md
are referenced below by relative path, and resolve inside this skill's own
directory (templates under assets/, the report generator under
scripts/). Where that directory IS on disk varies by run, so anything you
must invoke by absolute path is under $AEP_SKILLS_DIR/aep-validation/ —
the runner sets it.
Workflow
1. Read the issue
The prompt carries the issue URL. Read it with comments:
gh issue view <url> --comments
The body must contain: the acceptance oracle section (criteria file path + per-criterion tables), a Test layout section, and Report requirements. If a required section is missing, post an issue comment naming what's missing and exit with failure.
Deployed endpoint URLs and any test credentials are NOT in the issue — they are runtime inputs kept out of the public issue. The endpoints are already on disk for you (step 4); credentials you request on demand.
Post a brief opening comment (Starting validation: <one-line plan>).
2. Branch
The aep/m<milestone#>- prefix is a CONTRACT, not a style: the platform keys
your merged pull request back to this run by the branch name. A branch outside
that shape reads as a stranger's pull request — it is refused auto-merge, and
your tests and report never reach the run. The milestone is the one your
validation issue is filed under:
MILESTONE=$(gh issue view <N> --repo <owner/repo> --json milestone -q .milestone.number)
git checkout -b "aep/m${MILESTONE}-validation"
If MILESTONE comes back empty the issue was filed without one — a platform
fault, not something to work around. Say so in an issue comment and stop; a
branch without the prefix would strand everything you are about to do.
3. Partition the criteria
Read the criteria file (default
specs/validation/validation-criteria.json) — read-only. Coverage is
determined by committed-spec presence, not by any flag in the file.
For each criterion:
method: e2e, no spec attests/e2e/specs/<AC-ID>.spec.ts→ author a spec (steps 5–6).method: e2e, a committed spec already exists attests/e2e/specs/<AC-ID>.spec.ts→ regression set: do not re-author it, just run it (heal only perreferences/healing.md).method: manual→ no automation; the report renders these as a human checklist.method: scenario→ no automation in this run; the report lists them as not validated.
4. Read the validation context, then confirm the app is reachable
Deployed endpoint URLs come from the platform, never from the issue. The
platform fetched them before you started and left them at
/tmp/validation-context.json:
cat /tmp/validation-context.json
The content is { "endpoints": [{"component","url"}], "criteriaPath": "..." }. It is present and non-empty whenever you are running: the runner
exits before starting you if the platform could not resolve it, so there is
no failure mode here for you to recover from. If the file were somehow
missing, that is a platform fault — say so in one line and stop. Do not
probe, scan, or infer endpoints, and do not call the platform for them
yourself; the URL is not something you can work out from inside the cluster.
-
Endpoints become
tests/e2e/targets.json(step 5) and your target list. Probe each URL (curl -sf -o /dev/null <url>or a playwright-cli visit) before authoring. Never start, build, or deploy the app — you validate what is already running; an unreachable endpoint → issue comment + exit failure. -
Test credentials (on demand): request them only when a criterion needs a login — POST the test-credentials endpoint with an optional
rolehint (the role the flow requires).AEP_TASK_IDis this run's validation cycle id; the bearer rides a file:curl -sf -X POST "$AEP_PLATFORM_URL/internal/v1/validation/$AEP_TASK_ID/test-credentials" \ -H "Authorization: Bearer $(cat "$AEP_BEARER_FILE")" \ -H "Content-Type: application/json" \ --data '{"role":"admin"}' > /tmp/creds.jsonThe response is
{ "username", "password", "mock": true|false, "note": "..." }. Export it in-session and never commit it:export AEP_E2E_USERNAME=… AEP_E2E_PASSWORD=…. Whenmockis true the account is a shared stand-in (real user provisioning isn't implemented yet) — use it, and state in your PR description and closing comment that auth-gated criteria ran against mock credentials, since such a login may legitimately fail against a generated app that doesn't recognise it. Only if the request itself errors do you let the affected criterion landnot_run, blocker noted. -
Local dev servers (experimental runs only): if the fetched endpoints are
localhostdev servers you must start (the local harness), this overrides the base "never start servers" rule: start each per the repo's README, backgrounded with logs to a file (e.g.nohup go run . > /tmp/api.log 2>&1 &), then poll until it answers. These die with your session; never commit their artifacts or logs.
5. Scaffold the test harness (skip pieces that already exist)
The Playwright package lives at repo root tests/e2e/ — outside every
component's App Path, so committing tests never triggers a component
rebuild. Never touch application source or the root package.json.
tests/e2e/
package.json see below
package-lock.json generated by npm install; commit it
playwright.config.ts copy assets/playwright.config.template.ts
targets.json from the fetched validation-context endpoints (step 4)
lib/targets.ts copy assets/targets.template.ts
specs/ one spec file per criterion
scripts/generate-report.mjs copy the plugin's scripts/generate-report.mjs VERBATIM
.gitignore node_modules/ test-results/ playwright-report/ heal-log.json
-
package.json— exactly this shape (no"type": "module": the templates rely on Playwright's default CommonJS transpilation), with@playwright/testpinned to the exact value of the$AEP_PLAYWRIGHT_VERSIONenv var (the image's baked browsers match that version — a floating^would download browsers at run time):{ "name": "e2e", "private": true, "devDependencies": { "@playwright/test": "<value of $AEP_PLAYWRIGHT_VERSION>" } } -
targets.jsonshape:{"targets": {"<component>": "<url>", ...}, "primary": "<the web-facing component>"}, filled from the step-4 validation-contextendpoints. On a re-validation run, refresh it from the context file — a committedtargets.jsonmay name URLs from an earlier deployment. -
Install with
npm installon first scaffold (commit the lockfile),npm cion later runs. -
scripts/generate-report.mjsis platform-owned: the REPORT step always executes the plugin's copy directly and refreshes this committed copy, which exists only so humans can reproduce the report after checkout.
6. PLAN, then GENERATE
Before this step, pull in the phase discipline — do not proceed from memory:
- Read
references/authoring.mdnow and follow it as the binding authoring discipline (plan format, collect-generated-code loop, assertion rules, criterion↔spec contract). - Load
aep:playwright-cliwith the Skill tool — the CLI's own skill (commands, refs, eval, storage state; vendored from @playwright/cli).
Then: write the test plan, author one spec per uncovered e2e criterion (source-informed where unambiguous; playwright-cli exploration when the live app is the only trustworthy source), and remember the bar: a spec counts only after passing twice consecutively against the live app.
- Plan artifact:
tests/validation/test-plan.md— one section per criterion (id, must, target, numbered steps, expected assertion). Commit it before writing specs. On re-validation, append new sections; don't rewrite history. - Spec naming (the report's join key — get this exactly right):
- file:
tests/e2e/specs/<AC-ID>.spec.ts(e.g.AC-001-a.spec.ts) - title:
test('<AC-ID>: <short form of the must>', ...)
- file:
7. RUN
cd tests/e2e && npx playwright test
The config writes test-results/results.json. The run includes the
regression set — that's free regression coverage, not an accident.
8. HEAL (bounded)
For failures, Read references/healing.md now and follow it as the
binding HEAL discipline before touching any spec. Then: triage each
one against the live app, repair only brittleness (locators, waits,
setup), never weaken what a test asserts. Log each heal in
tests/e2e/heal-log.json. When the budget is exhausted, finish with
one final full run so results.json reflects the authoritative state.
9. REPORT
Generate the report deterministically — never hand-write it. Run the platform's copy of the script (always the current version, regardless of what an earlier cycle committed into the repo):
node "$AEP_SKILLS_DIR/aep-validation/scripts/generate-report.mjs" \
--issue <N> --commit "$(git rev-parse HEAD)"
Then refresh the repo's committed copy so a human can reproduce the report after checkout:
cp "$AEP_SKILLS_DIR/aep-validation/scripts/generate-report.mjs" \
tests/e2e/scripts/generate-report.mjs
This writes tests/validation/report.md + report.json. It reads the
oracle but never writes it — coverage is expressed by each criterion's
pass/fail in the report, not by a flag in the criteria file. Exit code 2
means a contract violation: spec titles that don't map to criterion ids
(fix titles, re-run tests from step 7), a spec file missing its
// spec: header (add the header, regenerate — no test re-run needed),
or a pre-existing spec modified without a heal-log entry (record the
heal per references/healing.md).
tests/validation/report.json is REQUIRED. Commit it with the tests.
The platform reads it at your pull request's merge commit to decide the
run's verdict. A merged PR without it fails the whole run with
validation-unreported — the platform cannot tell "nothing was wrong"
from "nobody looked", so it assumes nothing was learned. This is not
best-effort.
So on exit code 2: fix the violation and regenerate. Never open the PR without the report, and never hand-write one to get past the error — a fabricated report is worse than a failed run.
A failing criterion is different: that is report content, it belongs in the report, and you still open the PR (step 10).
10. PR
gh pr create \
--title "Validation: <pass>/<total> e2e criteria passing (issue #<N>)" \
--body $'Closes #<N>\n\n<summary table: pass/fail/not_run + manual/scenario counts>\n\nReport: tests/validation/report.md'
Open it ready-for-review even when criteria fail — the human reads the report and decides. Post a closing issue comment with the summary counts and the PR link.
Do not
- Modify application source, component App Paths, or the root
package.json— tests and report only. - Write or modify anything under
specs/— the acceptance oracle is read-only input; all validation artifacts live undertests/. - Delete or skip a previously committed spec. If one is obsolete, say so in the PR description and report — a human removes it.
- Leave
.only/.skip/.fixmein committed specs. - Hand-edit
report.md/report.json— regenerate via the script. - Commit playwright-cli session state, server logs,
test-results/, or credentials. Credentials come only from the step-4 test-credentials request (exported in-session asAEP_E2E_USERNAME/AEP_E2E_PASSWORD, never written to a file); if the request errors and a criterion needs login, mark it blocked in an issue comment and let it landnot_run. - Everything in the
aepskill's deny-list (no default-branch pushes, no force-push, one PR, no merging, no repo-settings changes).