moai-core | Based on revfactory/harness testing methodology single source of truth for: 4-dimensional scoring rubric, skill chain verification protocol
Overview
Verifies the quality of skills generated by skill-builder or existing skills. Implements harness's A/B testing methodology to compare baseline (no skill) and with-skill (using skill), and 4-dimensional scoring rubric과 chain verification protocolare included directly in the body so it operates standalone without referencing a separate rules file.
Trigger Keywords
skill test verification A/B test baseline regression test chain test skill-tester quality measurement rubric scoring 4-dimensional evaluation
Workflow
1. [로드] → 대상 스킬의 tests/test-cases.yaml 로드
2. [선택] → 테스트 모드 선택 (A/B | 회귀 | 체인 | 루브릭 단독)
3. [실행] → 테스트 프롬프트를 Claude Code에 실행
4. [측정] → 토큰 사용량, 출력 품질, 시간 측정
5. [스코어] → 4차원 루브릭 가중 평균 산출
6. [비교] → baseline vs with-skill 결과 비교
7. [보고] → 스코어 리포트 생성
Execution Rules
Test Modes
Mode 1: A/B Test (New Skill Verification)
This is the default mode used in skill-builder Phase 5.
Procedure:
- Baseline Execution: Run the same prompt in Claude Code without the skill
- With-Skill Execution: Run the same prompt after loading the skill
- Comparative Measurement:
| Metric | Measurement Method | Improvement Criterion |
|---|---|---|
| Token Usage | input + output tokens | -10% or more decrease |
| Output Quality | Rubric Score | +0.15 or more improvement |
| Accuracy | assertion pass rate | 80% or more |
| Completeness | Whether required output is included | 100% |
Sample Size Guide:
| Expected Improvement Rate | Minimum Sample Count |
|---|---|
| >= 20% | 2-3회 |
| 10-20% | 3-5회 |
| < 10% | 5 or more runs |
Mode 2: Regression Test (After Modifying an Existing Skill)
When modifying a skill, verify that existing test cases still pass.
Procedure:
- Load the last test result before modification as baseline
- Re-run the same test cases after modification
- Comparison: if a previously passing item changes to failing, it is a regression
Regression Determination:
| Change | Determination |
|---|---|
| Previously passing → still passing | PASS |
| Previously passing → failing | REGRESSION (modification rollback required) |
| Previously failing → passing | IMPROVEMENT |
| New test added | INFO (not comparable) |
Mode 3: Chain Test (Skill Combination Verification)
Tests a chain in which multiple skills are connected sequentially. All skill chains defined in the project are subject to verification.
3-1. Chain Definition Format
chain:
name: "blog-publishing"
description: "Blog post creation → AI slop review → optional media generation"
steps:
- skill: "moai-content:blog"
output_type: "markdown"
provides: ["blog_draft"]
- skill: "moai-core:ai-slop-reviewer"
input_from: ["blog_draft"]
output_type: "markdown"
provides: ["reviewed_draft"]
- skill: "moai-media:higgsfield-image"
input_from: ["reviewed_draft"]
output_type: "image_url"
optional: true
provides: ["cover_image"]
3-2. Test Design Rules
Happy Path Test — Run the entire chain with representative input and verify the following:
- Each step's output is compatible with the next step's input schema
- The final output matches the documented expectations
- No data loss between steps
Failure Propagation Test — Simulate a failure at each step:
- Does the chain halt gracefully?
- Is the user informed of which step failed?
- Can it resume from the point of failure?
Optional Step Test — Chains with optional steps:
- Does the chain complete even when an optional step is skipped
- Does the chain complete even when an optional step fails
- Is the result meaningful even without the optional step's output
Output Compatibility Test — At each step boundary:
- The output schema matches the next step's expected input
- Language consistency (Korean output → Korean input)
- Format consistency (markdown → markdown)
3-3. Test Case Template
## Chain Test: <chain-name>
### TC-1: Happy Path
- Input: <대표 프롬프트>
- Expected Step 1 output: <설명>
- Expected Step 2 output: <설명>
- Expected Final output: <설명>
- Pass criteria: 모든 단계가 기대 출력 생성, 최종 결과 사용 가능
### TC-2: Step <N> Failure
- Input: <단계 N 실패 유발 프롬프트>
- Expected behavior: <체인 정지 / 건너뜀 / 재시도>
- Pass criteria: 사용자에게 전달, 데이터 손실 없음
### TC-3: Optional Step Skip
- Input: <선택 단계 트리거 없는 프롬프트>
- Expected behavior: <선택 단계 없이 체인 완료>
- Pass criteria: 최종 출력이 선택 단계 기여 없이도 유의미함
3-4. Known Chains
| Chain | Steps | Status |
|---|---|---|
| Business plan (PPT) | strategy-planner → pptx-designer → ai-slop-reviewer | Subject to verification |
| Blog publishing | blog → ai-slop-reviewer → (optional) higgsfield-image | Subject to verification |
| Product landing | copywriting → landing-page → ai-slop-reviewer | Subject to verification |
3-5. Regression integration
When any skill in a chain is modified:
- Re-run all chain tests that include that skill
- Focus verification on step-boundary output compatibility
- When the output schema changes, update downstream skills accordingly
Mode 4: Rubric standalone (Standalone Scoring)
Performs 4-dimensional scoring using only the body of a specific skill's SKILL.md. Used when absolute evaluation is needed without A/B comparison.
Scoring rubric (4-dimensional weighted average)
All skills (new/modified) must be scored with this rubric before release.
Dimension definitions
| Dimension | Weight | Key question |
|---|---|---|
| Correctness | 30% | Does the output achieve its intended purpose? |
| Completeness | 25% | Does it handle edge cases and common variations? |
| Clarity | 25% | Can the user understand it and make use of the result? |
| Efficiency | 20% | Is token usage appropriate relative to output quality? |
Correctness — 30%
| Score | Description |
|---|---|
| 1.0 | Fully achieves the stated purpose without factual errors. All examples within the skill produce accurate results. |
| 0.75 | Achieves the purpose but with minor deviations in format or style. No factual errors. |
| 0.50 | Achieves most of the purpose but omits 1-2 key aspects or has minor inaccuracies. |
| 0.25 | Addresses only part of the purpose. Major gaps or errors exist. |
Completeness — 25%
| Score | Description |
|---|---|
| 1.0 | Handles all documented use cases. Covers edge cases. Documents error paths. Specifies fallback skill when not applicable. |
| 0.75 | Handles core use cases. Covers most edge cases. Error handling exists but is incomplete. |
| 0.50 | Main use cases work. Edge cases not covered. Minimal error handling. |
| 0.25 | Only the happy path works. No edge case handling. Silent failure on error. |
Clarity — 25%
| Score | Description |
|---|---|
| 1.0 | Executable without re-reading. Workflow is clear. Output format is predictable. Examples can be copied and pasted for use. |
| 0.75 | Understood after one read. Minor ambiguity in 1-2 steps. Examples are useful but need modification. |
| 0.50 | Requires re-reading sections. Some steps open to interpretation. Examples exist but are abstract. |
| 0.25 | Workflow is confusing. Steps are highly open to interpretation. No concrete examples. |
Efficiency — 20%
| Score | Description |
|---|---|
| 1.0 | Token usage optimized relative to output quality. No duplicate instructions. Progressive disclosure well utilized. Loads only when needed. |
| 0.75 | Token usage is reasonable. Minor duplication possible. Progressive disclosure mostly effective. |
| 0.50 | Token usage is high relative to need. Some sections can be shortened. Progressive disclosure not utilized. |
| 0.25 | Excessive token usage. Lots of boilerplate. No progressive disclosure. |
Pass Criteria
- Minimum Pass: Weighted average >= 0.70
- Single-dimension floor: Hard fail if any dimension is below 0.50
- New Skill: Requires >= 0.75 on first evaluation
Weighted Score Calculation
score = (correctness * 0.30) + (completeness * 0.25) + (clarity * 0.25) + (efficiency * 0.20)
Evaluation Procedure
- Self-evaluation: Skill author self-scores with this rubric
- Peer evaluation: A second evaluator scores independently (Complex tier recommended)
- Gap resolution: Discuss and resolve if per-dimension score difference > 0.15
- Final score: Average of all evaluators
Skill Tier Integration
| Skill Tier | Required Score | Evaluation |
|---|---|---|
| Simple (<50줄) | >= 0.70 | Self-evaluation |
| Standard (50-150줄) | >= 0.70 | Self + recommended peer |
| Complex (150줄+) | >= 0.75 | Self + required peer |
Anti-Pattern Checks (boundary-verification integration)
Before finalizing the pass score, confirm the following:
- Does not duplicate functionality covered by other skills
- No references to deprecated fields/patterns
- Values that should be configurable are not hardcoded
- Example outputs are real, not fabricated
- Triggers do not overlap with other skills without a clear priority
Test Case Format
For each skill, the tests/test-cases.yaml file format:
skill: <skill-name>
version: 2.27.0
test_cases:
- id: TC-001
name: "happy-path"
prompt: |
<사용자 프롬프트>
assertions:
- type: contains
value: "<출력에 포함되어야 할 문자열>"
- type: not_contains
value: "<출력에 포함되지 않아야 할 문자열>"
- type: format
value: "<markdown|json|text|html>"
quality_threshold:
correctness: 0.75
completeness: 0.70
clarity: 0.70
efficiency: 0.60
- id: TC-002
name: "edge-case"
prompt: |
<경계 조건 프롬프트>
assertions:
- type: handles_gracefully
value: true
Score Report Format
Report generated after testing is complete:
## Skill Test Report: <skill-name>
### Summary
- Mode: A/B Test
- Date: YYYY-MM-DD
- Result: PASS / FAIL
### Scores
| Dimension | Baseline | With-Skill | Delta |
|-----------|----------|------------|-------|
| Correctness | 0.65 | 0.85 | +0.20 |
| Completeness | 0.60 | 0.80 | +0.20 |
| Clarity | 0.70 | 0.80 | +0.10 |
| Efficiency | 0.75 | 0.80 | +0.05 |
| **Weighted** | **0.67** | **0.81** | **+0.14** |
### Token Usage
- Baseline: XXXX tokens
- With-Skill: XXXX tokens
- Delta: -XX%
### Assertions
- TC-001: PASS (3/3 assertions)
- TC-002: PASS (2/2 assertions)
### Anti-Pattern Audit
- [x] 중복 없음
- [x] 하드코딩 없음
- [x] 모든 예시 real
### Recommendation
<APPROVE for release / NEEDS revision on [dimension]>
Usage Examples
Example 1: A/B Testing a New Skill
"Run an A/B test of the sales-playbook skill with skill-tester"
Example 2: Regression Testing
"I modified the blog skill, run a regression test"
Example 3: Chain Testing
"Run the blog → ai-slop-reviewer chain test"
Example 4: Rubric-only Evaluation
"Evaluate the kr-gov-grant skill only with the 4-dimensional rubric"
Output format
| Deliverable | Format | Description |
|---|---|---|
| Score Report | Markdown | 4-dimensional rubric score + assertion results + anti-pattern audit |
| Test Results | YAML | Detailed results per test case |
Precautions
- A/B tests must be run on the same model under the same conditions
- Token usage is
/costchecked with the command - Automated CI integration is planned for implementation in a later Phase 4
- Do not include sensitive information (such as real customer names) in test prompts
- This SKILL.md is the single sourceof the 4-dimensional rubric and chain protocol.
.claude/rules/harness/quality/The file of the same name in is a redirect stub, and its body refers here.
Related skills
| Skill | Relationship | Description |
|---|---|---|
| skill-builder | before | Run tests after skill creation |
| skill-template | before | Define template-based skill structure |
| ai-slop-reviewer | alternative | Text quality review (non-functional) |
Related commands
| Command | Description |
|---|---|
/harness |
In the automatic new→test→review chain, this skill is called at the test stage |
Source: revfactory/harness skill-testing-guide + qa-agent-guide + Pipeline pattern (Apache 2.0) + MoAI adaptation