validate-tests

validate-tests -- Test Anti-Pattern Detection

Corresponding rule: testing.md

Purpose

Statically detect anti-patterns in test code based on behavior-driven testing principles (testing.md). Identify patterns that hurt maintainability, such as implementation-detail testing, excessive mocking, and shared global state.

Input

• Target: test files (*.test.*, *.spec.*, test_*, *_test.*)
• Default search path: entire project. If CLAUDE.md specifies a test directory, that path takes precedence.

Validation Categories

1. Vague Test Names

Detect patterns where test names provide no clue to the cause of failure.

Search patterns:

• test_\d+$ (names ending only in a number: test_1, test_2)
• test_it_works
• test_function
• test_ok
• test_success
• it\(["']works["'] (JS/TS: it('works'))
• it\(["']should work["']
• test\(["']test["'] (JS/TS: test('test'))

Severity: WARN

2. Implementation-Detail Tests

Detect patterns that verify call count/existence of internal methods.

Search patterns:

• .call_count
• .called
• .assert_called_with
• .assert_called_once
• .assert_called_once_with
• .assert_any_call
• .toHaveBeenCalledTimes (Jest)
• .toHaveBeenCalledWith (Jest)
• verify(.*).times( (Java Mockito)

Severity: WARN

3. Excessive Mocks

Detect patterns where a single test function uses 4 or more Mock objects.

Search patterns:

• Mock() or mock.patch appears 4+ times in a single test function
• jest.fn() or jest.mock appears 4+ times in a single test
• @mock.patch decorators: 4+ on a single test function

Severity: WARN (when 4 or more)

4. Shared Global State

Detect patterns where test files create instances at module level, sharing state across tests.

Search patterns:

• = new (instance creation) -- at the top level of a test file (outside functions/classes)
• Variables with shared_ prefix
• Module-level let declarations that are mutated by tests (JS/TS)
• Use of the global keyword at the top level of a test file

Severity: WARN

5. DB Dependency in Unit Tests

Detect patterns where unit test files directly call DB connections/queries. Exclude integration test files whose names contain integration, e2e, or integ.

Search patterns:

• db.query
• db.execute
• db.session
• db.connect
• prisma. (query methods)
• mongoose.connect
• connection.execute
• pool.query

Exclusion condition: file name contains integration, e2e, or integ

Severity: WARN

Execution Logic

1. Glob to collect test files (*.test.*, *.spec.*, test_*, *_test.*)
2. Categories 1, 2, 5: Grep each pattern across all test files
3. Category 3: Read test files and tally Mock count per function
4. Category 4: Read test files and detect instance creation at module level (outside functions/classes)
5. Group detected anti-patterns by category and output as a table

Mandatory Output: Test Validation Matrix

Output the matrix below before generating the final report. Do not proceed until every category has been scanned.

Category	Status	Items Checked	Findings	Severity	Evidence
Vague Test Names	?	?	?	WARN	{Glob/Grep, files, pattern}
Implementation-Detail Tests	?	?	?	WARN	{Glob/Grep, files, pattern}
Excessive Mocks	?	?	?	WARN	{Read, scanned files}
Shared Global State	?	?	?	WARN	{Read, scanned files}
DB Dependency	?	?	?	WARN	{Grep, files, pattern}

Status values: PASS (verification complete, no issues), NOT_APPLICABLE (no test files), SKIPPED (plugin issue), SHALLOW (partial scan)

Pre-Output Checklist (Mandatory Before Final Output)

Check every item before writing the report. If any item is unchecked, go back and complete it.

• Every category has a Status value (no empty Status cells)
• Every category with Status != SKIPPED has an Evidence value
• All 5 validation categories scanned (or marked NOT_APPLICABLE)
• Every finding includes file path and line number
• Legacy-allowance annotations applied where applicable
• Report language matches the user's conversation language

Schema Compliance Check (Mandatory Before Saving to .ww-w-ai/)

Before writing to .ww-w-ai/devtools/validate-tests/, verify the JSON output:

• Every "required" field in schema.json is present and non-empty
• The findings[] array contains every detected anti-pattern
• categorySummary counts match the actual finding counts per category

Output

Generate the validation report in the user's conversation language.

Output detection results as a Markdown table in this format:

===== Test Validation Report =====

| File:Line | Type | Description |
|-----------|------|-------------|
| `src/services/auth.test.ts:15` | Vague Name | `test_1` -- cannot identify failure cause |
| `src/services/order.test.ts:42` | Implementation Detail | `.assert_called_once_with` -- internal-call verification |
| `src/services/payment.test.ts:20` | Excessive Mocks | 6 Mocks used -- design review needed |
| `test_user.py:5` | Global State | `shared_cart = Cart()` -- state shared across tests |
| `src/utils/calc.test.ts:30` | DB Dependency | Direct `db.query` call -- unit test depends on DB |

Total anti-patterns: {N}
  Vague Names: {A}
  Implementation Detail: {B}
  Excessive Mocks: {C}
  Global State: {D}
  DB Dependency: {E}

===============================

When no anti-patterns are found:

===== Test Validation Report =====

No anti-patterns.

===============================

Output Persistence

After generating the test validation report, save results to .ww-w-ai/devtools/validate-tests/:

1. Create .ww-w-ai/devtools/validate-tests/ if missing
2. Write latest.json -- structured result following templates/schema.json
3. Write latest.md -- human-readable report following templates/report.template.md
4. Archive to history/ -- copy latest.json to .ww-w-ai/devtools/validate-tests/history/{timestamp}.json

latest.md is generated in the user's conversation language. JSON field names stay in English regardless of language. The JSON output enables machine-parseable history tracking and cross-run comparison. The history/ directory preserves previous executions for trend analysis.

Permission Rationale

• Write: Exclusively for .ww-w-ai/ output persistence. No project source modification.
• Bash: Exclusively for read-only git/system queries. No file modifications.

Notes

• Bash is permitted for directory creation (mkdir -p .ww-w-ai/devtools/validate-tests/history). The validation itself is read-only and does not modify project files.
• "Change-detection tests" in legacy code are temporarily allowed per testing.md exceptions. If the file or directory contains legacy, migration, or compat, annotate the result with "(legacy allowance)."
• Implementation-detail test detection (Category 2) may yield false positives. .toHaveBeenCalledWith can also be used for external-dependency verification (e.g., API calls), so results are presented as advisory.
• Excessive-Mocks judgment (Category 3) is performed per function. Mocks set up in beforeEach/setUp are distributed and tallied against each test.

Spec Reference

For detailed validation criteria, evidence tables, and examples:

• Corresponding rule spec: ../../docs/specs/testing.md