academic skill eval team

Academic Skill Eval Team — Skill Multi-Agent Evaluation Team

For a single skill or an entire skill pack organize a "multi-agent evaluation team" that gives structured evaluation conclusions across six dimensions: trigger words, instruction quality, workflow completeness, output specification, robustness, and packaging consistency.

Core Philosophy

Evaluation is not simply "praising whether it's good or not"; rather, it answers 4 questions:

1. Will this skill be triggered correctly?
2. Once triggered, is the execution flow complete, clear, and actionable?
3. Is the output specific, stable, and reusable?
4. As a publishable skill, are its packaging and collaboration chain consistent?

Judgments must be based on repository evidence; vague evaluations are prohibited.

Agent Team Composition

Role	Focus	Required Actions
Agent 1: Trigger Auditor	Trigger words and invocation timing	Check manifest.json and SKILL.md whether their trigger scenarios are consistent, whether they cover both Chinese and English, and whether they are prone to false triggering/missed triggering
Agent 2: Workflow Auditor	Instruction structure and execution flow	Check whether the skill contains clear steps, boundary conditions, input requirements, and connections with other skills
Agent 3: Output Judge	Output templates and deliverability	Check whether the output format is stable, whether the information is sufficient, and whether it can be directly used by downstream processes
Agent 4: Red-Team Tester	Robustness and failure modes	Design boundary inputs, ambiguous requests, and information-missing requests to determine whether the skill will go off track, over-speculate, or miss key constraints
Agent 5: Pack Integrator	Whole-package consistency	Check manifest.json、README.md、CHANGELOG.md, whether the descriptions of each skill are consistent, and evaluate whether the workflow coordination is smooth

If the user evaluates only a single skill, still keep the 5 roles, but focus the emphasis on the target skill; if the user evaluates the entire pack, give conclusions skill by skill and add a system-level evaluation.

Workflow

Step 1: Clarify the evaluation target

First identify whether what the user wants to evaluate is:

• A single SKILL.md
• a certain skill directory
• the entire skill pack
• a new draft/not-yet-registered skill

If the user does not specify a target, by default evaluate "the most relevant skill in the current repository or the entire pack".

Step 2: Collect evaluation evidence

At minimum, check the following materials:

• manifest.json
• The target skill's SKILL.md
• README.md
• CHANGELOG.md(if present)
• examples/、docs/ examples in that are strongly related to the target skill (if present)

The following three categories of issues must be distinguished:

1. Skill prompt design issues: missing processes, unclear templates, unclear boundaries
2. Packaging/registration issues: inconsistencies in manifest, version, trigger words, paths
3. Workflow collaboration issues: difficulty connecting the input and output between upstream and downstream skills

Step 3: Conduct a 5-agent division-of-labor review

Agent 1: Trigger Auditor

Key checks:

• manifest.json Whether trigger_phrases covers natural expressions
• SKILL.md whether the description accurately describes the trigger conditions
• Whether overly broad trigger words cause false activation
• Whether overly narrow trigger words make it hard to discover
• Whether Chinese and English triggering is balanced

Agent 2: Workflow Auditor

Key checks:

• Is there a clear step-by-step process?
• Are input prerequisites defined, and how to handle missing information?
• Are the default strategy, optional branches, and stopping conditions explained?
• Does it tell the model to analyze first before outputting, rather than generating conclusions directly?
• Can it be chained with other skills?

Agent 3: Output Judge

Key checks:

• Is there a standard output structure?
• Are the output fields sufficient for users to use directly?
• Are there clear tables, templates, or checklists?
• Can the output be consumed by downstream skills?
• Does it avoid giving only vague suggestions?

Agent 4: Red-Team Tester

Construct at least 3 types of challenge inputs:

1. Vague request: the user's goal is incomplete, with only one sentence given
2. Request with missing information: missing key inputs such as papers, images, repositories, color schemes, etc.
3. Mixed request: multiple goals mixed together, prone to going off-topic

For each type of input, judge:

• How the skill will respond
• Whether hallucinated completion will occur
• Whether necessary clarification will be skipped
• Whether non-executable results will be output

Agent 5: Pack Integrator

Key checks:

• manifest.json、README.md、SKILL.md whether the version and description are consistent
• Whether the new skill fits into the existing workflow
• Whether the skill's purpose needs to be exposed in the README
• Whether there are inconsistencies in paths, naming, or terminology
• Whether the entire pack forms a clear "input → analysis → output" loop

Step 4: Scoring

Score each dimension on a scale of 1-5 points, and explain the reasoning:

Dimension	Description
Discoverability	Whether the trigger words and skill positioning are clear and easy to discover
Instruction Quality	Whether the instructions are specific, structured, and unambiguous
Workflow Completeness	Whether the inputs, steps, branches, and boundary conditions are complete
Output Usefulness	Whether the output is stable, specific, and directly deliverable
Robustness	Whether it is robust to vague/missing/mixed inputs
Pack Consistency	Whether it is consistent with the manifest, README, and upstream/downstream skills

Scoring guidance:

• 5 = ready to publish, with only minor optimization items
• 4 = generally reliable, with a few medium-priority improvements
• 3 = usable but unstable; key issues should be fixed before release
• 2 = many issues, prone to false triggering or unstable output
• 1 = obvious structural defects; not recommended for release

Step 5: Form a conclusion

The final output must provide:

1. Overall rating:可直接发布 / 小修后发布 / 需要重构后再评测
2. Total score and sub-scores
3. Main findings of each agent
4. Top 3 High-priority modification suggestions
5. Challenge prompts that can be directly re-tested

Output format

Strictly output according to the following template:

---

Skill Eval Report

1. Scope

• Evaluation target：
• Evaluation scope: single skill / entire pack
• Evidence sources：

2. Overall Verdict

• Conclusion: ready to publish / publish after minor revisions / needs refactoring before evaluation
• Total score：X/30
• One-sentence summary：

3. Scorecard

Dimension	Score	Conclusion	Evidence
Discoverability	X/5
Instruction Quality	X/5
Workflow Completeness	X/5
Output Usefulness	X/5
Robustness	X/5
Pack Consistency	X/5

4. Agent Findings

Agent 1: Trigger Auditor

• Strengths:
• Issues:
• Suggestions:

Agent 2: Workflow Auditor

• Strengths:
• Problems:
• Suggestions:

Agent 3: Output Judge

• Strengths:
• Problems:
• Suggestions:

Agent 4: Red-Team Tester

• Challenge input 1:
• Challenge input 2:
• Challenge input 3:
• Main risks:

Agent 5: Pack Integrator

• Consistency issues:
• Workflow issues:
• Suggestions:

5. Priority Fixes

6. Retest Prompts

---

Quality standards

A high-quality evaluation report must satisfy:

• All conclusions are supported by repository evidence
• Can distinguish between "problems with the skill itself" and "problems with the packaged documentation"
• Not only point out problems, but also give actionable fix suggestions
• Can point out the user input scenarios most likely to fail
• Applicable to both a single skill and the entire pack

Notes

• Do not give a high score just because the content is long; focus on whether it is actionable
• Do not equate a well-written README with a well-designed skill
• Do not ignore boundary conditions just because the examples look nice
• If version inconsistencies, trigger word inconsistencies, or workflow breaks are found, they must be clearly pointed out
• If the evidence is insufficient, clearly write "insufficient evidence"; do not speculate

Relationship with other skills

This skill is not responsible for generating paper figure content, but for evaluating the quality of other skills. It is suitable for use in the following scenarios:

• Conduct a structured review before a new skill is released
• Perform regression checks on existing skills
• Check whether the collaboration loop of the entire pack is complete
• Discover issues with trigger words, README, version, output templates, etc. before open-source release