automate-tests

Skill: Run Automated Tests

Purpose

Work out how the target repository expects automated tests to be run (commands, frameworks, prerequisites, and scope), then run the best-matching test suite under a safety-first interaction policy.

Core Objective

Primary goal: produce test execution results through evidence-based command selection and safety guardrails.

Success criteria (all must hold):

✅ Test plan discovered: the evidence source is identified (documentation, CI configuration, or build manifest)
✅ Command selected: the appropriate test command is chosen for the mode (fast/ci/full) and the constraints
✅ User confirmation obtained: approval received before installing dependencies, using the network, or starting services
✅ Tests executed: the command was run with its output and exit code captured
✅ Results summarized: a test plan summary with evidence, commands, execution status, and failures (if any)

Acceptance test: can a developer reproduce the test execution by following the test plan summary, with no extra context?

Scope Boundaries

This skill owns:

Discovering the test command from repository evidence (documentation, CI, build manifests)
Selecting the appropriate test command for the mode and the constraints
Executing tests behind safety guardrails and user confirmation
Summarizing test results with evidence and failure diagnostics

This skill does not own:

Test quality assessment or coverage analysis (use review-testing)
Fixing failing tests or debugging test failures (use orchestrate-repair-loop)
Writing new tests or test infrastructure (use the development skills)
Reviewing test code against best practices (use review-testing)

Handoff point: once the tests finish (pass or fail), hand off to orchestrate-repair-loop to fix the failures, or to review-testing for a quality assessment.

Use Cases

You cloned a repository and want the right test command without guessing.
The repository has several test layers (unit/integration/e2e) and you need a safe default run plan.
CI failed and you want to reproduce it locally by running the same commands the workflow uses.

Behavior

Establish scope and constraints (ask when unclear)
- With nothing specified by the user, default to a fast, local, non-destructive run:
  - Unit tests only, no external services, no Docker, no network-dependent setup.
- Where it matters, have the user pick a mode:
  - fast: unit tests only, minimal setup.
  - ci: mirror the CI workflow commands as closely as possible.
  - full: include integration/e2e tests and service dependencies.
- Ask whether Docker is allowed, whether network access is allowed, and whether dependency installation is allowed.
Discover the test plan (evidence-based)
- If CLAUDE.md or .ai-cortex/config.yaml exists, prefer the test_command recorded there; otherwise discover it from the sources below. See docs/guides/project-config.md.
- Read these sources in order; stop early once a clear, unambiguous test command turns up:
  - README.md, CONTRIBUTING.md, TESTING.md, docs/testing*, Makefile
  - CI configuration: .github/workflows/*.yml, .gitlab-ci.yml, azure-pipelines.yml, Jenkinsfile
  - Build manifests: package.json, pyproject.toml, setup.cfg, tox.ini, go.mod, pom.xml, build.gradle*, *.csproj, Cargo.toml
- Identify:
  - The primary test entry point (npm test, pnpm test, yarn test, pytest, tox, go test, dotnet test, mvn test, gradle test, cargo test, and so on)
  - Test layers and markers (unit, integration, e2e)
  - Environment prerequisites (DB, Redis, Docker Compose, required environment variables, secrets)
  - How CI sets up its dependencies (services, caches, artifacts)
- Prefer an explicit statement in the documentation or CI over heuristic inference.
Choose the execution plan
- In ci mode: derive the run sequence from the repository's CI workflow steps (closest match).
- In fast mode: pick the most direct unit test command, the one with the fewest prerequisites.
- With several stacks present (backend + frontend, say), suggest running each stack separately in a fixed order.
- If the plan needs a dependency install or a service start, ask for confirmation before continuing.
Execute with guardrails
- Always print the exact command you are about to run, before running it.
- Use a working directory rooted at the target repository (default .).
- Capture and summarize failures:
  - The first failing command and its exit code
  - The most relevant error excerpt
  - Next actions (missing toolchain, missing environment variable, service not running, and so on)
- Avoid destructive operations:
  - Do not run rm -rf, git clean -fdx, docker system prune, or database drop/migration commands without the user's explicit approval.
- If the repository needs secrets, do not ask the user to paste them into the chat. Use a .env file, a secret manager, or the documented local development flow.

Input & Output

Input

Target repository path (default .).
Mode: fast (default), ci, or full.
Constraints: dependency install allowed (yes/no), network allowed (yes/no), Docker allowed (yes/no).

Output

A short "test plan summary" containing:
- Evidence: which files/paths informed the plan
- The selected commands (in order)
- Assumptions and prerequisites
- What was executed and what was skipped (and why)
Enough of the command log to debug a failure (do not dump extremely long logs unless asked).

Restrictions

Hard Boundaries

Do not invent a test command when evidence exists (documentation/CI takes precedence).
Do not install dependencies, run Docker, or start external services without confirmation.
Do not modify repository files unless the user explicitly asks (exception: generating a report file when the user asks for an artifact).
Do not leak secrets; do not request sensitive credentials in the chat.

Skill Boundaries (avoid overlap)

Do not do these (other skills handle them):

Test quality assessment: judging test coverage, test design, or testing best practices → use review-testing
Fixing test failures: debugging failing tests, repairing broken test code, or root-causing → use orchestrate-repair-loop
Writing tests: creating new test cases, test infrastructure, or a test framework → use the development/implementation skills
Code review: reviewing test code for quality, maintainability, or best practices → use review-testing
Repository analysis: full codebase structure analysis or architecture review → use review-codebase

When to stop and hand off:

Tests fail and the user asks "why?" or "how do I fix it?" → hand off to orchestrate-repair-loop for debugging and repair
The user asks "are these tests any good?" or "what is our coverage?" → hand off to review-testing for a quality assessment
The user asks "can you write tests for X?" → hand off to the development workflow for test implementation
Tests pass and the user asks "what should we test next?" → hand the test strategy suggestion to review-testing

Self-Check

Core success criteria (all must hold)

Test plan discovered: the evidence source is identified (documentation, CI configuration, or build manifest)
Command selected: the appropriate test command is chosen for the mode (fast/ci/full) and the constraints
User confirmation obtained: approval received before installing dependencies, using the network, or starting services
Tests executed: the command was run with its output and exit code captured
Results summarized: a test plan summary with evidence, commands, execution status, and failures (if any)

Process quality checks

Evidence-based selection: did I identify at least one authoritative source for the test command (a doc file, a CI workflow, or a build manifest)?
Safety guardrails applied: did I ask for confirmation before installing dependencies, using the network, starting Docker/services, or any state-changing action?
Commands printed: did I print the exact command before running it?
Failures diagnosed: when tests failed, did I give the first failing command, the exit code, and a likely root-cause category?
No destructive operations: did I avoid running destructive commands (rm -rf, git clean, docker system prune, database drops) without explicit approval?
No secret leakage: did I avoid asking for sensitive credentials in the conversation, using a .env file or the documented local development flow instead?

Acceptance test

Can a developer reproduce the test execution by following the test plan summary, with no extra context?

If no: the test plan summary is incomplete. Add the missing evidence, commands, or prerequisites.

If yes: the skill run is complete. Continue to the handoff if one is needed.

Examples

Example 1: a JavaScript repo with package.json

User: "run the tests for this repository."

Agent:

Check the package.json scripts and .github/workflows/*.
Settle on fast mode and propose:
- npm test (or pnpm test / yarn test if the repository standardizes on one)
Ask: "install dependencies (npm ci) and allow network access?"
Run:
- npm ci
- npm test
Summarize the results and point at the failing test output, if any.

Example 2 (edge case): a monorepo that needs Docker for integration tests

User: "mirror CI locally."