SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsqa-test-script-generator

qa-test-script-generator

Testing & quality

View repository

Updated 10-16 09:34

About this skill

QA Test Script Generator — script generation

CLI invocation convention: The CLI of this toolkit is quality-assurance-agent/scripts/qa_agent.py. Itis not distributed as a PATH command—the command is executed by you (the agent); a human does not need to type it by hand. Resolve the skill directory once before starting work; after that, all commands are written as python "$QA_AGENT_DIR/scripts/qa_agent.py" <cmd>：

QA_AGENT_DIR="${QA_AGENT_CLI:-$(dirname "$(find ~/.claude/skills ~/.agents/skills ~/.codex/skills .claude/skills .agents/skills .codex/skills -maxdepth 2 -name SKILL.md -path '*quality-assurance-agent/*' 2>/dev/null | head -1)")}"

If the runtime environment has already been told this skill directory (Claude Code will), use it directly; no need to run the lookup above. For the complete command syntax see $QA_AGENT_DIR/references/cli-reference.md。

Your positioning

You are the conversion layer between use cases and execution.qa-testcase-designer What [it] produces is use cases for humans to read; what you produce is tasks that the execution phase runs. If your spec-task is not written precisely, the later execution phase cannot reliably judge pass/fail.

Youdo not executetasks,do not mark pass/fail、do not judge readiness. Implementation status only goes as far as implemented, execution status is left to not-run。

CLI commands

For the complete syntax and parameter descriptions of all commands in this phase, seemain skill (quality-assurance-agent) → CLI command reference → Phase 3. Command syntax is not duplicated here.

Generation hierarchy convention

The script layer follows thecomplete test pyramid（generate-spec-tasks default behavior): each use case is decomposed level by level in decreasing order of "unit > integration > API > E2E." Acceptance must likewise have unit tests (testing white-box behavior such as fund calculations, state machines, validation logic, etc.)—do not cut any layer down to 0.

Default minimum task count: P0=8, P1=5, P2=3, P3=1; default ratio unit 60% / integration 20% / api 15% / e2e 5%. Pure backend scope (no frontend UI) automatically skips E2E.

Workflow

1. Validate use cases

First run validate-cases to ensure the use cases themselves have no schema problems. Only use cases in the confirmed status generate tasks.

2. Generate spec-task

First read $QA_AGENT_DIR/references/spec-task-planning.md to understand the spec-task field contract, coverage rules, and completion gate rules.

Run generate-spec-tasks. Ensure:

  • Each task has a oracle.ui/api/db/sideEffects/negativeAssertions field
  • Each P0/P1 risk (from risk-analysis.json) appears in the assertions or oracle of the corresponding task

3. Validate coverage

Run coverage-balance --strict. If it fails, fix the spec-task plan rather than forcing it to pass.

3.5 Validate script implementation authenticity (mapping ≠ implementation)

python "$QA_AGENT_DIR/scripts/qa_agent.py"assert-script-implementation --spec-tasks .qa-agent/current/test-spec-tasks.json --repo .

Gate validation: for each targetFile the file exists, is not a placeholder stub, and the number ofdiscoverable test methods ≥ the number of tasks mapped to it。

  • One targetFile maps 54 tasks, but there are only 7 @Test methods → gate fail (inflated implementation).
  • Placeholder stub scripts (echo "BLOCKED"; exit 0) → gate fail (not actually implemented).

4. Implement test files

Implement test scripts item by item according to the spec-task:

  • api layer task: bash script (curl + python3 parsing JSON assertions) or JUnit integration test class, directly calling the backend API
  • integration layer task: for those requiring database verification, script + MySQL MCP read_query combined
  • e2e layer task: spec file path is tests/e2e/<module>/<case-id>.spec.ts（@playwright/test format), execution command is npx playwright test <targetFile>. The spec file content is generated by the Playwright Test Agent (planner → generator) or written by hand. Team-shared fixtures are in tests/e2e/lib/e2e-fixture.js（init-project auto-deployed), providing loginAsQA() and other reusable operations
  • Reuse existing tests: precisely map targetFile/testName/command/assertions/oracle/evidence to spec-task

General conventions for test scripts:

  • Use python "$QA_AGENT_DIR/scripts/qa_agent.py"run-with-env --repo . --script <path> to execute, automatically loading environment variables
  • Input parameters are passed via --extra KEY=VAL (do not hardcode passwords and paths)
  • Exit code 0 = pass, non-0 = fail
  • After each task is implemented, update implementationStatus=implemented，executionStatus to keep not-run

5. Required fields for writing a spec-task

Each task must include:

  • sourceCaseId——traceable to the test case
  • priority——must be consistent with the corresponding test case (otherwise assert-completion will filter it out and report a filter diagnostic)
  • layer——unit / integration / api / e2e
  • targetFile——test file path
  • testName——readable test name
  • command——command that can be copy-pasted and executed
  • assertions——derived from test case businessAssertions , deduplicated (template placeholders must not be used)
  • oracle——ui / api / db / sideEffects / negativeAssertions categories,each item is a structured object {"type","assertion","sourceRiskId"?,...}, not a string
  • traceability——list of associated risk IDs
  • dataBindings——script parameters derived from test case data (hardcoding business IDs is prohibited)
  • implementationStatus——implemented or not-implemented
  • executionStatus——yours here is always not-run, left for the execution phase to change

Root-level fields:generationProfile(fixed to development, i.e. the full pyramid),minSpecsByPriority、targetRatio——coverage-balance will read this metadata to automatically match the ratio.

oracle structured contract

{
  "type": "db",
  "assertion": "t_user.virtual_usd_balance 减少 totalDeducted",
  "sourceRiskId": "RISK-P0-001",
  "tableHint": "t_user"
}
  • A DB oracle must contain specific query objects and expectations; generalized wording such as "core state is consistent" is not allowed.
  • Data integrity test cases (such as probability table gap validation) generate verificationMode: direct-db tasks, and oracle.db must be non-empty.
  • requiresE2E=true risk, even if e2e is squeezed to 0 in the ratio, at least one e2e task must still be forcibly generated.

E2E layer output conventions

Uniformly use the @playwright/test format, generating spec files via the Playwright Test Agent (planner → generator) or by hand.

Item Convention
targetFile tests/e2e/<module>/<case-id>.spec.ts(lowercase ID, e.g. tc-p1-011.spec.ts）
command npx playwright test <targetFile>
shared fixture require('../lib/e2e-fixture')（init-project auto-deploys e2e-fixture.js）
configuration source config/env.shared(team-shared) → local/.env(personal secrets)
browser driver @playwright/test（test()、expect()、page fixture）

E2E fixture（tests/e2e/lib/e2e-fixture.js) is first deployed by init-project , providing loginAsQA()、getTestValue() and other reusable operations. The team can maintain this fixture together later.

Fault tolerance and degradation

  • Upstream artifact missing：test-cases.json does not exist or is unconfirmed, fall back to qa-testcase-designer phase, no task is generated.
  • Use case schema validation failed：validate-cases , does not pass → send back for correction, do not force-generate spec-task.
  • coverage-balance failed: fix the spec-task plan rather than force-passing the gate.
  • encoding corruption：test-spec-tasks.json after finishing writing, you must run check-mojibake --strict。U+FFFD → safe-write-json to rewrite.
  • test file write conflict: when the path already exists and has content, first read the existing file, do incremental merging rather than blindly overwriting.

Prohibitions

  • Do not mark tasks as passed/failed. Only execution can do this.
  • Do not reduce the number of tasks just to pass completion。
  • Do not bypass the P0/P1 minimum task count constraints and risk-derived oracle requirements。
  • Do not equate the overall passing of an existing test suite with spec-task mapping. There must be a precise file/command/assertion mapping.
  • Do not create excessive E2E tasks, unless the scope explicitly configures E2E coverage. Maintain an inverted pyramid of unit > integration > api > e2e.
  • Do not say the work is done just because "generation is finished". Your work starts only after the upstream use cases are confirmed, and ends when spec-tasks and test files are in place. The completion judgment is not yours to make.
  • Generating placeholder stub scripts is prohibited. No script may exist in the form of echo "BLOCKED: ..."; exit 0 . If data preparation truly cannot be automated (requires destructive DB operations, etc.), you should mark the task in test-spec-tasks.json as blocked, and clearly write blocker/owner/nextAction; do not use a fake script to cover up the blocked state.

DB Assertion Solidification and MCP Data Preparation Specification

DB assertions must not be "outsourced" to the execution runner for manual verification—they must be solidified into executable structured checks. A bash script cannot directly call MySQL MCP (MCP is Claude's tool, not a shell command), so there are two categories:

Data integrity use cases (direct-db)

For "data integrity" type use cases (such as probability table gap validation), generate verificationMode: direct-db tasks,oracle.db must contain "specific SQL + expected values", to be executed one by one by the runner with results recorded, not treated as a free-form manual verification item.

{
  "verificationMode": "direct-db",
  "oracle": {
    "db": [
      {
        "type": "db",
        "assertion": "每个概率表首条 outcome_from=0，末条 outcome_to=1",
        "query": "SELECT product_id, MIN(outcome_from) f, MAX(outcome_to) t FROM t_product_prize GROUP BY product_id",
        "expect": "f=0 且 t=1.0",
        "sourceRiskId": "RISK-P1-005"
      }
    ]
  }
}

API call type use cases (data preparation PRE/POST)

The script contains the complete API call + assertion logic; data preparation is annotated with comments, to be completed by the runner via MCP before and after the script. Data preparation isinput preparation, DB assertions must still go into oracle.db structured fields.

#!/usr/bin/env bash
# Test: TC-P2-032 - Exchange软删除
# MCP 数据准备（执行 runner 在脚本前后完成）：
#   PRE:  mcp__mysql_mcp__write_query "UPDATE t_user_open_record SET is_deleted=1 WHERE id=<PENDING_ID>"
#   POST: mcp__mysql_mcp__write_query "UPDATE t_user_open_record SET is_deleted=0 WHERE id=<SAME_ID>"
set -u
# ... login, API call, assertions ...

Error patterns (prohibited)

#!/usr/bin/env bash
echo "BLOCKED: Requires data preparation"
exit 0  # ← 永远禁止！这是在伪造"执行过"

Responsibilities of the execution runner

qa-test-runner When encountering verificationMode: direct-db tasks or scripts annotated with PRE:/POST: :

  1. Execute PRE data preparation via MySQL MCP
  2. Run the script (API calls + assertions)
  3. Execute via MCP oracle.db verification one by one, recording "query summary + expected + actual + status" to evidence
  4. Execute POST data restoration via MCP
  5. All MCP operation failures are also recorded in evidence; an empty oracle.db is considered an incomplete spec-task and must not enter the execution phase