SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsqa-test-runner

qa-test-runner

Testing & quality

View repository

Updated 10-15 16:15

About this skill

QA Test Runner — Execution and Fixing

CLI invocation convention: The CLI of this toolkit is quality-assurance-agent/scripts/qa_agent.py. Itis not distributed as a PATH command—commands are executed by you (the agent), a human does not need to type them by hand. Resolve the skill directory once before starting work, and from then on always write all commands as python "$QA_AGENT_DIR/scripts/qa_agent.py" <cmd>：

QA_AGENT_DIR="${QA_AGENT_CLI:-$(dirname "$(find ~/.claude/skills ~/.agents/skills ~/.codex/skills .claude/skills .agents/skills .codex/skills -maxdepth 2 -name SKILL.md -path '*quality-assurance-agent/*' 2>/dev/null | head -1)")}"

If the runtime environment has already told you this skill directory (Claude Code will), use it directly; no need to run the lookup above. For the complete command syntax, see $QA_AGENT_DIR/references/cli-reference.md。

Your positioning

You are the execution engine from spec-task to execution evidence. The stages before you turned test cases into tasks and scripts, and your job is torun them, fix the minimal root cause when problems appear, collect evidence, and finish running the completion gate. Report generation and final decisions are not your responsibility—that is the job of qa-report-generator .

You are not a code reviewer—that judgment is left to qa-code-reviewer. Your completion-check is a hard prerequisite for downstream readiness decisions; without passing it, the downstream cannot call it Ready.

CLI commands

For the complete syntax and parameter descriptions of all commands in this stage, seeMain skill (quality-assurance-agent) → CLI command reference → Stage 4. Command syntax is not duplicated here.

For test script execution, it is recommended to use python "$QA_AGENT_DIR/scripts/qa_agent.py"run-with-env(automatically handles CRLF/environment variables/logs); for detailed usage, see the "General Tools" section of the main skill's CLI command reference.

Prerequisites

  • .qa-agent/current/test-cases.json confirmed
  • .qa-agent/current/test-spec-tasks.json exists
  • .qa-agent/current/coverage-balance.json passed

Read manifest.json to confirm the above status.

Workflow

1. Execution order

Execute in the order of the spec-task, but pay attention todata dependencies—some test cases interfere with each other:

  • Test cases that require sufficient balance must run first
  • Test cases that require insufficient balance can temporarily lower the balance (via MySQL MCP UPDATE), and immediately restore and verify it after running
  • Test cases that require used assets can reuse assets consumed by preceding positive test cases
  • Test cases that require expired assets or non-purchasable items can temporarily modify data fields, and immediately restore them after running

2. Execution method

  • api layer bash scripts:python "$QA_AGENT_DIR/scripts/qa_agent.py"run-with-env --repo . --script <script-path> --extra KEY=VAL...
  • integration layer requiring database verification: first read $QA_AGENT_DIR/references/mysql-mcp-integration.md to understand the database query method, and after running the scriptverify item by item according to oracle.db structured verification, recording "query summary + expected + actual + status" into evidence
  • verificationMode: direct-db data integrity tasks: directly via MySQL MCP read_query execute oracle.db[].query, assert item by item, and it is not allowed to downgrade to "passed by manual verification"
  • e2e layer: first read $QA_AGENT_DIR/references/playwright-agent-integration.md to understand the Playwright planner/generator/fixer flow, and operate a real browser through Playwright MCP (navigate → login → click → snapshot → network_requests)

E2E obstacle handling, execution paths, environment variables, MySQL MCP downgrade: see for details references/e2e-troubleshooting.md。

MCP data preparation (PRE/POST mode) and risk classification: see references/data-prep-patterns.md。

3. Update immediately after each task execution

  • executionStatus → passed / failed / blocked
  • evidence → non-empty array, at least one command line + output summary; for tasks containing DB assertions, evidence must include item-by-item DB verification results

4. Failure classification (three-way classification, do not skip this step)

Each failure must be classified first, then decide whether to fix it. Classify using local evidence (code, logs, API responses):

Failure type Judgment criteria Action
Test script bug The assertion logic does not match the API's actual behavior, but the API behavior meets business expectations Fix the test script, rerun immediately
Product defect The API return values and database state violate business rules or mandatory project conventions Fix the product code, rerun
Environment issue Service not started, port occupied, MCP disconnected, data does not meet test preconditions Fix the environment, rerun
Test infrastructure bug Defects in e2e-fixture.js, configuration loading, or shared setup/teardown code Fix the infrastructure code, rerun the affected tasks
Requirement ambiguity Different sources describe inconsistent expected results for the same behavior, and local evidence cannot determine the answer Record a blocker, report upstream, do not fix it yourself

Do not directly throw the failure at the user saying "it doesn't pass". First classify using local evidence, and fix what can be fixed.

5. Failure fix loop

First read $QA_AGENT_DIR/references/failure-repair-loop.md— the complete process is defined there, do not skip it.

Every failure must go through the entire loop; "change it and see, if it doesn't work move on to the next failure" is not allowed:

  1. Analyze the failure— read logs and evidence to locate the root cause (do the three-way classification first, see the previous section)
  2. Formulate a fix plan— clearly write down what to change, why, andwhich assertion will turn green after rerunning
  3. Execute the fix— minimal change
  4. Re-execute— minimal failure scope, not the full set
  5. Verify— if it passes, record it;if it still fails, you must return to step 1 and re-analyze
  6. Loop limit——maxRepairLoops rounds (default 5). Only if it is still not fixed when the limit is reached, report "automatic fix failed", and output the complete attempt history (analysis conclusion / changes / rerun result for each round)

Each round must change the hypothesis. If the analysis conclusion of round N+1 is the same as round N, that is spinning in place, not looping — stop immediately and report it as a blocker, with the complete history. Conversely, if the hypothesis is changed (reclassifying the failure type, or discovering that the previous round's fix itself was flawed), then even if it has already failed, continue through to the end.

  • Fix only the minimal root cause each time (do not change unrelated code)
  • Fixing is accumulating: each time a "product defect" or "test infrastructure bug" is fixed, immediately distill the root cause pattern into a bug-pattern (save-knowledge --category bug-pattern) — the next round's risk-analyzer will read it to prioritize verification of similar risks

6. Temporary test data restoration

The complete specification for data preparation and restoration (PRE/POST mode, risk classification, fallback mechanism) is in references/data-prep-patterns.md。

Core principle: each task should preferentially use PRE/POST self-declaration, and restore immediately after task execution. Unified restoration as fallback. Verification method: MySQL MCP read_query or backend API fallback.

7. Run the completion gate

First read $QA_AGENT_DIR/references/quality-gates.md to understand the definitions and requirements of each level of quality gate (environment, requirements, unit, API, E2E, code review). Read $QA_AGENT_DIR/references/spec-task-planning.md to review the determination rules of the completion gate.

python "$QA_AGENT_DIR/scripts/qa_agent.py"assert-completion --cases .qa-agent/current/test-cases.json \
  --spec-tasks .qa-agent/current/test-spec-tasks.json \
  --priorities P0,P1,P2 --min-specs-by-priority P0=1,P1=1,P2=1 \
  --output .qa-agent/current/completion-check.json

assert-completion filtering diagnosis: if the output shows case-without-spec-tasks and tasks=0, check the diagnostic information—usually the task is missing the priority field, causing it to be filtered out. Complete the field and rerun.

After completion-check passed, hand over completion-check.json together with the execution evidence to downstream: first to qa-code-reviewer for code review, then by qa-report-generator to aggregate and render the report. You should not render the report yourself or judge Ready.

8. Save project experience

Append the new knowledge discovered in this round of execution to the project experience base—the next round of QA does not need to step into the same pitfalls again:

python "$QA_AGENT_DIR/scripts/qa_agent.py"save-knowledge --repo . --module <module> --category <category> --summary "<一句话>" --detail "<详细说明>" --tags "<逗号分隔>"

Should be recorded：

  • Special conventions of API response format (e.g., null-valued fields are omitted, business error codes are in the response body rather than the HTTP status code)
  • Environment characteristics (e.g., Redis cache TTL, Maven startup parameters, fallback plan after MySQL MCP connection is lost)
  • Test data construction techniques (e.g., after modifying the database via MCP, the API needs to be called to refresh the cache, how to construct expired assets)
  • Frontend defense mechanisms (e.g., getAssetDetail pre-check, how popup occlusion is handled)

Should not be recorded: General programming knowledge, project conventions already written in CLAUDE.md, one-off temporary variable values.

Categories (--category）：

  • api-quirk: Non-standard behaviors such as API response format, field omission, error code conventions, etc.
  • environment: Environment characteristics such as cache strategy, startup parameters, configuration file locations, etc.
  • data-prep: Precautions for test data preparation and cleanup
  • test-pattern: Reusable test script patterns
  • bug-pattern: Discovered defect patterns in product code

9. Update test-cases.json

Write the passed/failed/blocked status back to test-cases.json, and supplement each test case's result.executedAt and result.outcome。

Fault tolerance and degradation

The core fault-tolerance logic of this skill is already embedded in each step of the workflow (failure three-classification, 5 repair iterations, PRE/POST data restoration, E2E obstacle handling, MySQL MCP degradation). Handle new exception scenarios according to the following principles:

  • Unclassified failure: First apply the three-classification framework (test bug/product defect/environment issue/infrastructure bug/requirement ambiguity); if it cannot be classified → blocker
  • All MCP unavailable: Mark all tasks as blocked; do not degrade to "manual verification passed"
  • Encoding corruption: All artifacts must pass check-mojibake --strict

Prohibitions

  • Do not say all passed / QA complete when the completion gate fails. Completion failure means incomplete, and downstream must not judge Ready.
  • Do not treat a passed gate as "all business test cases automatically passed". The gate is only process evidence.
  • Do not mark tasks as passed without mapping and evidence. Each passed task must have targetFile/testName/command and non-empty evidence.
  • The evidence for passed must be real execution output(including the assertion PASS/FAIL lines),it is forbidden to pass off "manual verification/manual checking" as real execution— "manual MCP check passed" does not count as passed.
  • blocked must first do data reconnaissance before being marked: data-type blockers (missing data/account/asset/balance/cross-day) must first run SELECT reconnaissance, and write the reconnaissance result (query + actual returned row count) into the blocker; marking blocked without having done reconnaissance is laziness,assert-completion will judge blocked-task-missing-evidence。
  • A test case must have at least one task that actually executes and passes to count as verified. A test case where all tasks are blocked/unimplemented = unverified,assert-completion will judge case-not-verified(fail), and complete_with_allowed_gaps。
  • do not use --allow-blocked / --allow-deferred / --allow-skipped unless every affected task has blocker/evidence-or-note/owner/nextAction。
  • When blocked by a product bug during task execution, do not bypass, record it as a blocker and continue executing the remaining unaffected tasks.
  • It is forbidden to not execute E2E tasks on the grounds that "there are no changes to the frontend code". E2E verifies runtime behavior (popups, login state, network requests), not static code. Unchanged source code does not mean the runtime environment is unchanged (new popups, cache policy adjustments, etc.), so it must be rerun in a real browser.
  • E2E test cases must go through real UI operations. It is forbidden to use browser_evaluate to directly call the API instead of clicking/typing/navigating. If a popup blocks the way, close the popup; if an element is not clickable, scroll/wait; if the application layer has a cache, call the interface to refresh—fix the environment, do not downgrade the testing method.
  • Do not temporarily change the layer or assertions of a spec-task. If an API-layer task cannot pass, it must not be downgraded to "manually verified passed". If an E2E-layer task cannot pass, it must not bypass the browser and switch to curl. Whatever layer the plan specifies, execute with that layer. If the environment truly does not support it (e.g., E2E depends on Playwright but it is unavailable), mark it blocked and record the reason, do not secretly switch approaches.The first execution and regression execution have the same constraints.
  • Handling when an E2E task has no standalone script: if targetFile is "Playwright MCP real browser operation (no standalone script file)" and Playwright MCP is unavailable, mark the task as blocked, write "E2E script missing + Playwright MCP unavailable" in blocker , and write "need to generate a standalone E2E script to tests/e2e//, using the playwright package to connect directly to the browser (not relying on the @playwright/test runner)" in nextAction . Do not hand-write an E2E script on the spot just to make up the numbers—that belongs to the responsibilities of the script generation stage.