SuperAPI AI Directory

meta-skill-tester

General & other

View repository

Updated 10-15 21:34

About this skill

Skill Tester — Skill Quality Verification Tool (Rubric + Chain Built In)

moai-coworker | based on revfactory/harness testing methodology single source of truth for: 4-dimensional scoring rubric, skill chain verification protocol

Overview

Verifies the quality of skills generated by meta-skill-builder or existing skills. It implements harness's A/B testing methodology to compare baseline (no skill) and with-skill (using the skill), and 4-dimensional scoring rubric과 chain verification protocolare included directly in the body so it operates standalone without referencing a separate rules file.

Trigger Keywords

skill test verification A/B test baseline regression test chain test meta-skill-tester quality measurement rubric scoring 4-dimensional evaluation

Workflow

1. [로드]    → 대상 스킬의 tests/test-cases.yaml 로드
2. [선택]    → 테스트 모드 선택 (A/B | 회귀 | 체인 | 루브릭 단독)
3. [실행]    → 테스트 프롬프트를 Claude Code에 실행
4. [측정]    → 토큰 사용량, 출력 품질, 시간 측정
5. [스코어]  → 4차원 루브릭 가중 평균 산출
6. [비교]    → baseline vs with-skill 결과 비교
7. [보고]    → 스코어 리포트 생성

Execution Rules

Test Modes

Mode 1: A/B Test (New Skill Verification)

This is the default mode used in meta-skill-builder Phase 5.

Procedure:

  1. Baseline Run: Run the same prompt in Claude Code without the skill
  2. With-Skill Run: Run the same prompt after loading the skill
  3. Comparative Measurement:
Metric Measurement Method Improvement Criterion
Token Usage input + output tokens -10% or more decrease
Output Quality Rubric Score +0.15 or more improvement
Accuracy assertion pass rate 80% or more
Completeness Whether required output is included 100%

Sample Size Guide:

Expected Improvement Rate Minimum Sample Count
>= 20% 2-3회
10-20% 3-5회
< 10% 5 or more runs

Mode 2: Regression Test (After Modifying an Existing Skill)

When modifying a skill, verify that existing test cases still pass.

Procedure:

  1. Load the last test result before modification as the baseline
  2. Re-run the same test cases after modification
  3. Comparison: if a previously passing item changes to failing, it is a regression

Regression Determination:

Change Determination
Previously passing → still passing PASS
Previously passing → failing REGRESSION (modification rollback required)
Previously failing → passing IMPROVEMENT
New test added INFO (not comparable)

Mode 3: Chain Test (Skill Combination Verification)

Tests a chain in which multiple skills are connected sequentially. All skill chains defined in the project are subject to verification.

3-1. Chain Definition Format
chain:
  name: "blog-publishing"
  description: "Blog post creation → AI slop review → optional media generation"
  steps:
    - skill: "moai-marketer:content-blog"
      output_type: "markdown"
      provides: ["blog_draft"]
    - skill: "moai-coworker:ai-slop-reviewer"
      input_from: ["blog_draft"]
      output_type: "markdown"
      provides: ["reviewed_draft"]
    - skill: "moai-media:media-higgsfield-image"
      input_from: ["reviewed_draft"]
      output_type: "image_url"
      optional: true
      provides: ["cover_image"]
3-2. Test Design Rules

Happy Path Test — Run the entire chain with representative input and verify the following:

  • Each step's output is compatible with the next step's input schema
  • The final output matches the documented expectations
  • No data loss between steps

Failure Propagation Test — Simulate a failure at each step:

  • Does the chain halt gracefully?
  • Is it communicated to the user which step failed?
  • Can it resume from the point of failure?

Optional Step Test — A chain with optional steps:

  • Does the chain complete even when an optional step is skipped
  • Does the chain complete even when an optional step fails
  • Is the result meaningful even without the optional step's output

Output Compatibility Test — At each step boundary:

  • The output schema matches the next step's expected input
  • Language consistency (Korean output → Korean input)
  • Format consistency (markdown → markdown)
3-3. Test Case Template
## Chain Test: <chain-name>

### TC-1: Happy Path
- Input: <대표 프롬프트>
- Expected Step 1 output: <설명>
- Expected Step 2 output: <설명>
- Expected Final output: <설명>
- Pass criteria: 모든 단계가 기대 출력 생성, 최종 결과 사용 가능

### TC-2: Step <N> Failure
- Input: <단계 N 실패 유발 프롬프트>
- Expected behavior: <체인 정지 / 건너뜀 / 재시도>
- Pass criteria: 사용자에게 전달, 데이터 손실 없음

### TC-3: Optional Step Skip
- Input: <선택 단계 트리거 없는 프롬프트>
- Expected behavior: <선택 단계 없이 체인 완료>
- Pass criteria: 최종 출력이 선택 단계 기여 없이도 유의미함
3-4. Known Chains
Chain Steps Status
Business plan (PPT) consult-strategy → doc-pptx → ai-slop-reviewer Subject to verification
Blog publishing content-blog → ai-slop-reviewer → (optional) media-higgsfield-image Subject to verification
Product landing content-copywriting → marketing-landing-page → ai-slop-reviewer Subject to verification
3-5. Regression integration

When any skill in a chain is modified:

  1. Re-run all chain tests that include that skill
  2. Focus verification on step-boundary output compatibility
  3. When the output schema changes, update downstream skills accordingly

Mode 4: Rubric standalone (Standalone Scoring)

Performs 4-dimensional scoring using only the SKILL.md body of a specific skill. Used when absolute evaluation is needed without A/B comparison.

Scoring rubric (4-dimensional weighted average)

All skills (new/modified) must be scored with this rubric before release.

Dimension definitions

Dimension Weight Key question
Correctness 30% Does the output achieve its intended purpose?
Completeness 25% Does it handle edge cases and common variations?
Clarity 25% Can the user understand and make use of the result?
Efficiency 20% Is token usage appropriate relative to output quality?

Correctness — 30%

Score Description
1.0 Fully achieves the stated purpose without factual errors. All examples within the skill produce accurate results.
0.75 Achieves the purpose but with minor deviations in format/style. No factual errors.
0.50 Achieves most of the purpose but omits 1-2 key aspects or has minor inaccuracies.
0.25 Addresses only part of the purpose. Major gaps or errors exist.

Completeness — 25%

Score Description
1.0 Handles all documented use cases. Covers edge cases. Documents error paths. Specifies alternative skills when not applicable.
0.75 Handles core use cases. Covers most edge cases. Error handling exists but is incomplete.
0.50 Main use cases work. Edge cases are not covered. Minimal error handling.
0.25 Only the happy path works. No edge case handling. Silent failure on error.

Clarity — 25%

Score Description
1.0 Executable without rereading. Workflow is clear. Output format is predictable. Examples can be used by copy-paste.
0.75 Understandable after one read. Minor ambiguity in 1-2 steps. Examples are useful but require modification.
0.50 Requires rereading sections. Some steps have room for interpretation. Examples exist but are abstract.
0.25 Workflow is confusing. Large room for interpretation in steps. No concrete examples.

Efficiency — 20%

Score Description
1.0 Token usage is optimal relative to output quality. No redundant instructions. Progressive disclosure is well utilized. Loads only when needed.
0.75 Token usage is reasonable. Minor redundancy possible. Progressive disclosure is mostly effective.
0.50 Token usage is high relative to need. Some sections can be shortened. Progressive disclosure is not utilized.
0.25 Excessive token usage. Boilerplate 大. Lack of progressive disclosure.

Passing criteria

  • Minimum pass: weighted average >= 0.70
  • Single-dimension floor: if any dimension is below 0.50, hard fail
  • New skill: requires >= 0.75 on first evaluation

Weighted score calculation

score = (correctness * 0.30) + (completeness * 0.25) + (clarity * 0.25) + (efficiency * 0.20)

Evaluation procedure

  1. Self-evaluation: skill author self-scores using this rubric
  2. Peer evaluation: second evaluator scores independently (Complex tier recommended)
  3. Gap resolution: if per-dimension score difference > 0.15, discuss and resolve
  4. Final score: average of all evaluators

Skill Tier integration

Skill Tier Required score Evaluation
Simple (<50줄) >= 0.70 Self-evaluation
Standard (50-150줄) >= 0.70 Self + recommended peer
Complex (150줄+) >= 0.75 Self + required peer

Anti-Pattern Checks (boundary-verification integration)

Before finalizing the passing score, verify the following:

  • Does not duplicate functionality covered by another skill
  • No references to deprecated fields/patterns
  • Values that should be configurable are not hardcoded
  • Example output is real and not fabricated
  • Triggers do not overlap with other skills without clear priority

Test Case Format

For each skill, tests/test-cases.yaml File format:

skill: <skill-name>
version: 0.1.0
test_cases:
  - id: TC-001
    name: "happy-path"
    prompt: |
      <사용자 프롬프트>
    assertions:
      - type: contains
        value: "<출력에 포함되어야 할 문자열>"
      - type: not_contains
        value: "<출력에 포함되지 않아야 할 문자열>"
      - type: format
        value: "<markdown|json|text|html>"
    quality_threshold:
      correctness: 0.75
      completeness: 0.70
      clarity: 0.70
      efficiency: 0.60

  - id: TC-002
    name: "edge-case"
    prompt: |
      <경계 조건 프롬프트>
    assertions:
      - type: handles_gracefully
        value: true

Score Report Format

Report generated after testing is complete:

## Skill Test Report: <skill-name>

### Summary
- Mode: A/B Test
- Date: YYYY-MM-DD
- Result: PASS / FAIL

### Scores

| Dimension | Baseline | With-Skill | Delta |
|-----------|----------|------------|-------|
| Correctness | 0.65 | 0.85 | +0.20 |
| Completeness | 0.60 | 0.80 | +0.20 |
| Clarity | 0.70 | 0.80 | +0.10 |
| Efficiency | 0.75 | 0.80 | +0.05 |
| **Weighted** | **0.67** | **0.81** | **+0.14** |

### Token Usage
- Baseline: XXXX tokens
- With-Skill: XXXX tokens
- Delta: -XX%

### Assertions
- TC-001: PASS (3/3 assertions)
- TC-002: PASS (2/2 assertions)

### Anti-Pattern Audit
- [x] 중복 없음
- [x] 하드코딩 없음
- [x] 모든 예시 real

### Recommendation
<APPROVE for release / NEEDS revision on [dimension]>

Usage examples

Example 1: New skill A/B test

"A/B test the collab-sales-playbook skill with meta-skill-tester"

Example 2: Regression test

"I modified the content-blog skill, run a regression test"

Example 3: Chain test

"Run the content-blog → ai-slop-reviewer chain test"

Example 4: Rubric standalone evaluation

"Evaluate the consult-gov-grant skill using only the 4-dimensional rubric"

Output Format

Deliverables Format Description
Score Report Markdown 4-dimensional rubric score + assertion results + anti-pattern audit
Test Results YAML Detailed results per test case

Precautions

  • A/B tests must be run with the same model and under the same conditions
  • Token usage is /cost checked with the command
  • Automated CI integration is scheduled to be implemented later in Phase 4
  • Do not include sensitive information (such as real customer names) in test prompts
  • This SKILL.md is the single sourceof the 4-dimensional rubric and the chain protocol. .claude/rules/harness/quality/ The file with the same name in is a redirect stub, and its body refers here.

Related Skills

Skill Relationship Description
meta-skill-builder before Run tests after skill creation
meta-skill-template before Define template-based skill structure
ai-slop-reviewer alternative Text quality review (non-functional)

Related Commands

Command Description
/harness Call this skill at the test stage in the automatic new→test→review chain

Source: revfactory/harness skill-testing-guide + qa-agent-guide + Pipeline pattern (Apache 2.0) + MoAI adaptation