skill-creator-advanced

Skill Creator Advanced

The goal of this skill is to turn "making a skill" into a repeatable engineering process, rather than a one-off rambling about prompts.

It also provides:

An actionable process: from requirements, design, validation, evals, benchmarks, and packaging to iteration
Reusable scripts: initialization, format checking, validation, test plan generation, workspace preparation, benchmark aggregation, regression gate checking, packaging
Splittable reference documents: put long content into references/ to maintain progressive disclosure
A lightweight review viewer: organize with-skill / baseline results into reviewable HTML

Quick start (you only need to make one new skill)

First organize 2-3 use cases from the existing conversation, repo, or example tasks, and ask follow-up questions only if that is not enough.
For each use case, write a trigger phrase and done looks like.
Create the skill folder:

python scripts/init_skill_advanced.py <skill-name> --path <output-dir>

Complete the new skill's SKILL.md, prioritizing writing the YAML's description。
Perform format and structure checks:

python scripts/format_check.py <path/to/skill>
python scripts/quick_validate.py <path/to/skill>

Plan real test cases, and generate a test plan when necessary:

python scripts/generate_test_plan.py <path/to/skill> --out references/test_plan.md

Prepare the eval workspace so that with-skill / baseline can reuse a fixed directory structure:

python scripts/prepare_eval_workspace.py <path/to/skill>

Package into .skill：

python scripts/package_skill.py <path/to/skill> <output-dir>

If you want to optimize the trigger quality of the description, additionally prepare a trigger eval set, then run:

python scripts/run_eval.py --eval-set <path/to/trigger-evals.json> --skill-path <path/to/skill> --model <model-id>
python scripts/run_loop.py --eval-set <path/to/trigger-evals.json> --skill-path <path/to/skill> --model <model-id> --apply-best

How to operate

When the user wants to create or revise a skill, proceed in the following order; you may skip steps, but clearly explain why.

Phase 0: Extract requirements from context

First look at the conversation history, existing files, and existing processes, then decide whether to ask follow-up questions.
Communicate using terminology the user is familiar with; if the other person is not familiar with technical terms, do not assume jargon.
First determine whether it is really worth making into a skill: if the problem is one-off and has no reusable process, you should directly point out that it is not worth packaging as a skill.

Phase 1: Requirements and use cases

First obtain 2-3 concrete use cases.
Each use case must have at least: trigger phrase, required input, main steps, output, done looks like.
If the user only gives a vague goal, proactively propose a set of reasonable use cases for the other person to confirm.

Phase 2: Architecture and SKILL.md

Decide which content should go in scripts/、references/、assets/。
description must answer two things at the same time: what this skill does, and under what circumstances it should trigger.
description should use real user phrases, not the author's self-indulgent categorization.
Keep the core process in SKILL.md, and move details and variants to references/。

Phase 3: Write instructions

Prioritize writing instructions that change behavior; do not explain common sense that the model already knows.
If you can give a direct command, give a direct command; only add a sentence of why when the reason can reduce misuse.
By default, follow least surprise: make the skill's behavior match ordinary users' intuition, and do not secretly change the goal.

Phase 4: Format checking and minimal compliance validation

First run format_check.py to fix structural and formatting issues.
Then run quick_validate.py to perform a minimal compliance confirmation.

Phase 5: Testing, evals, and benchmarks

Triggering tests: should trigger, near-synonym rewrites, near-miss, should not trigger.
Multilingual tests: consider at least zh、en、mixed, abbreviations/colloquial names.
Skill overlap tests: list nearby skills that are easily confused and negative triggers.
Functional tests: Given/When/Then, including at least a happy path, edge case, and failure mode.
Write the approved test prompts into assets/evals/evals.json。
First establish <skill-name>-workspace/iteration-N/, with each eval having its own with_skill/ and baseline directory.
If the environment supports subagents or parallel workers, with-skill and baseline/old-skill should be launched in the same round; if not supported, they can be run sequentially, but keep the same directory structure.
Performance comparison: compare rounds, tool calls, failure rate, and result quality against the baseline.
ROI comparison: confirm whether the improvement is worth the extra token, time, and maintenance costs.
After running, use scripts/aggregate_benchmark.py to consolidate the benchmark, then use scripts/generate_review.py to generate the review viewer.
Use scripts/check_regression_gates.py to check whether the release threshold is met.
Test wording should be close to what real users would say; do not only test textbook-style prompts.

Phase 6: Packaging and Release

Use package_skill.py to generate .skill。
When sharing, README, installation instructions, and release notes should be placed outside the skill folder.

Phase 7: Iteration and Maintenance

Under-trigger: add real trigger phrases, proper nouns, and file types.
Over-trigger: add negative triggers, narrow the scope, and remove ambiguous wording.
Unstable execution: add validation and move fragile steps into scripts.
Content too large: shorten SKILL.md and move details down into references.
During iteration, prioritize collecting concrete failure cases and user feedback; do not change wording based only on gut feeling.

For full details, see:

references/lifecycle.md
references/testing-playbook.md
references/description-optimization.md
references/eval-workflow.md
references/eval-schemas.md
references/multilingual-trigger-strategy.md
references/skill-boundary-management.md
references/regression-gates.md
references/skill-roi-model.md
references/distribution-playbook.md
references/patterns-troubleshooting.md

Core Rules (please strictly follow)

Get the description right first

This is the main factor in whether the skill will be loaded.
The description should include real trigger phrases, work scenarios, and file types when necessary.
Prioritize making obvious queries hit reliably, then handle corner cases; do not write the description too broadly for the sake of a few odd sentences.

Learn from context first, then ask the fewest questions

First read the conversation, files, and existing skills.
Only ask the user follow-up questions when a high-risk assumption would skew the result.

Move fragile steps into scripts

Any check or transformation that is repetitive, error-prone, or needs to be deterministic should be written as a script.

Avoid context bloat

SKILL.md Put the process and navigation in.
Put details in references/, and read them only when necessary.

Test with realistic cases, not just polished examples

Use prompts close to actual conversation.
Compare against the baseline to confirm the skill actually helps, rather than just adding a bunch of instructions.

with-skill and baseline should be compared using the same batch of evals

Launch them in the same round as much as possible to avoid too much difference in timing and context conditions.
If revising an existing skill, the baseline should be the old version skill snapshot, not "no skill at all".

Handle skill boundaries first, then handle wording

If multiple skills compete for the same type of query, first create an overlap matrix and define in-scope / out-of-scope.
Do not try to force-resolve conflicts just by writing a longer description.

A skill whose ROI does not hold up is not worth keeping around

If the improvement is too small, the cost is too high, or maintenance is too heavy, directly consider narrowing the scope, splitting the skill, or falling back to a general prompt.

Do not put README.md in the skill folder

A README is for people to read and should be placed at the repo root or somewhere else outside the skill folder.

Writing and design guidelines

Describe things in language the user understands; do not assume they know your internal terminology.
Instructions should preferably start with a clear verb, for example "check first" or "if it fails, stop and report".
When the reason for a step can prevent errors, write the reason out; otherwise keep it concise.
A skill should not secretly change the task. If the workflow requires trade-offs, state the trade-off principles explicitly.
If a task actually should not be made into a skill, point out the reason directly instead of forcing content together.

Scripts you can use

scripts/init_skill_advanced.py: Create a SKILL.md skeleton with test/release fields.
scripts/format_check.py: Format and structure checker (including --fix）。
scripts/quick_validate.py: Minimal compliance validation.
scripts/generate_test_plan.py: Generate a test plan template.
scripts/prepare_eval_workspace.py: From assets/evals/evals.json create an iteration workspace.
scripts/aggregate_benchmark.py: Aggregate with-skill / baseline run results and output benchmark.json and benchmark.md。
scripts/check_regression_gates.py: Determine whether a release is possible based on the benchmark and threshold settings.
scripts/run_eval.py: Run the description trigger eval and output query-level and run-level diagnostic results.
scripts/improve_description.py: Rewrite the description based on the trigger eval failure patterns, preserving the transcript.
scripts/run_loop.py: Chain eval and description rewriting into multiple rounds of iteration, with the option to directly apply the best description.
scripts/generate_report.py: Generate an HTML report for description optimization.
scripts/utils.py: Shared SKILL.md / JSON read-write helpers.
scripts/package_skill.py: After validation, package into .skill。

Viewer / eval structures you can use

assets/evals/evals.json: Store real test prompts, expected outputs, and expectations.
assets/evals/regression_gates.json: Store the release threshold settings for the benchmark.
scripts/generate_review.py: Output workspace results as review HTML.
<skill-name>-workspace/iteration-N/: Store the output, grading, and benchmark for each round of with-skill / baseline.

Common deliverables

When delivering to the user, it usually includes:

skill folder（SKILL.md + scripts/ + references/ + assets/）
assets/evals/evals.json
<skill-name>-workspace/iteration-N/ benchmark and review output
.skill package file
README, examples, release notes, and installation instructions placed outside the skill folder