skill-creator

Skill Creator — SAP Skill Lifecycle

Primary reference: https://cap.cloud.sap/docs SAP UI5 docs: https://ui5.sap.com SAP Help Portal: https://help.sap.com/docs Skills live in: skillOverflow/skills/<skill-name>/SKILL.md (flat — category set via metadata.category)

A skill is not documentation. Documentation is written once and decays. A skill improves through real usage — every project session is an opportunity to encode what the agent got wrong into a permanent correction.

---

Stage 1: Identify the need

A skill is needed when you notice the agent repeatedly producing a suboptimal pattern. One mistake is noise. The same mistake across multiple sessions or developers is a signal.

Triggers that indicate a missing or outdated skill:

• Agent uses a deprecated CAP API (e.g. cds.service.impl, @sap/cds-mtx)
• Agent produces a known anti-pattern (e.g. key ID : UUID instead of cuid)
• Agent ignores a team convention consistently
• A new CAP version shipped and the agent doesn't know the new patterns yet
• A code review keeps catching the same issue — the agent and the reviewer disagree

Before creating a new skill, check:

ls ~/skillOverflow/skills/

The pattern might already be covered by an existing skill that just needs updating. Updating is almost always better than adding — fewer skills = less context overhead.

---

Stage 2: Author the skill

File location

skillOverflow/skills/<skill-name>/SKILL.md

Use kebab-case. Be specific: fiori-draft not fiori, cds9-migration not migration.

Required frontmatter

---
name: your-skill-name
description: >
  Use when [specific trigger conditions — be precise].
  Covers [what it teaches]. Trigger keywords: [terms a developer
  would actually type in their prompt].
---

The description is the most critical field. It's the only part the agent reads for every single message. If it's vague, the skill never triggers. If it's too broad, it triggers on everything and pollutes unrelated tasks.

Test your description by asking: "Would this description match if a developer typed X?" — where X is your target prompt. If yes, good. If no, rewrite.

Skill body structure

# Skill Name — CAP Best Practices

> **Primary reference**: https://cap.cloud.sap/docs/[relevant-page]

## [Pattern or concept name]

[One sentence: why this matters in practice]

\`\`\`cds
// ✅ Correct pattern — copy-pasteable, not pseudocode
entity Products : cuid, managed {
  title    : localized String(111);
  currency : Currency;
}
\`\`\`

## [Next pattern]

...

## Common mistakes to avoid

- ❌ [Specific bad pattern] — [why it fails]
- ✅ [Correct replacement]

- ❌ [Another bad pattern]
- ✅ [Correct replacement]

Writing rules

Do:

• Use real, copy-pasteable code examples — not pseudocode
• Cover one clear path that works well — not every possible variation
• Keep it under 500 lines — split into reference files if it grows beyond that
• Add a capire doc link at the top of the body
• End with a "Common mistakes to avoid" section with ❌/✅ pairs
• Write for the agent, not for a human reader — be explicit and unambiguous

Don't:

• Try to cover every edge case — opinionated and focused beats comprehensive
• Repeat what capire already says well — link to it instead
• Use vague language like "consider" or "you might want to" — be directive
• Include patterns that conflict with each other — pick one and commit

---

Stage 3: Write evals

Evals are your regression tests. They verify the skill produces better output than the agent would without it.

What makes a good eval

{
  "id": 1,
  "skill": "your-skill-name",
  "name": "descriptive name of what is being tested",
  "prompt": "A realistic prompt a developer would actually type — not crafted to match the skill",
  "assertions": [
    "Specific, observable thing the output must contain or do",
    "Another concrete assertion — no vague 'output is better' criteria"
  ]
}

Critical rule: evals must not be biased toward the skill.

If your eval prompt uses the exact same wording as your skill instructions, it will always win against a no-skill baseline — even if the output is identical. A good eval tests the outcome, not whether the agent repeated your instructions.

❌ Biased eval prompt (mirrors skill wording):

Create an entity using the cuid aspect and managed aspect from @sap/cds/common

✅ Unbiased eval prompt (how a developer would actually ask):

Add a Suppliers entity. It needs a name, country, and can have many Contracts.

Baseline comparison — the real test

Run your eval prompt with and without the skill:

# Disable the skill temporarily for the agent you're testing
mv ~/.copilot/skills/your-skill ~/.copilot/skills/your-skill.disabled

# Run the prompt → save output A

# Re-enable
mv ~/.copilot/skills/your-skill.disabled ~/.copilot/skills/your-skill

# Run the same prompt → save output B

# Compare against assertions

The skill earns its place only if output B meaningfully passes more assertions than output A. If both outputs are identical, the skill isn't adding value — simplify or remove it.

Add evals to the test suite

Add your eval to tests/skillOverflow-evals.json following the existing format. Minimum 2 evals per skill — one for the happy path, one for an edge case or a common mistake the skill is specifically designed to prevent.

---

Stage 4: Check and iterate

Run the smoke test after every skill change:

bash ~/skillOverflow/tests/smoke-test.sh ~/skillOverflow

Then run your evals manually using tests/skill-test-checklist.html.

When a skill underperforms — diagnosis questions:

Symptom	Likely cause	Fix
Skill never triggers	Description too vague	Add specific trigger keywords
Skill triggers on wrong prompts	Description too broad	Narrow the trigger conditions
Skill loads but output ignores it	Instructions too abstract	Add concrete code examples
Skill produces wrong patterns	Content outdated	Update to latest capire docs
Skill conflicts with another skill	Overlapping descriptions	Merge or split responsibilities

Testing across models:

When a new model version is released, re-run the full eval suite. Models improve over time — they may have learned CAP patterns that your skill was compensating for. When a model now produces the correct output without the skill, that section of the skill can be simplified or removed. Skills should shrink as models improve, not grow indefinitely.

---

Stage 5: Learn from real usage (the most important stage)

At the end of a real working session, ask the agent:

Summarise this session: what patterns did you get right on the first try,
where did you have to correct yourself, what assumptions turned out to be wrong,
and what would you add to the CAP skills to prevent those mistakes next time?

Then take the agent's answer and encode the corrections into the relevant skill.

What to capture:

• A pattern the agent had to retry → add it to "Common mistakes to avoid"
• An assumption the agent made that was wrong → add a clarification or guardrail
• A CAP API the agent reached for that doesn't exist → add the correct alternative
• A step the agent skipped that caused a problem later → make it explicit in the instructions

What not to capture:

• One-off mistakes that didn't repeat — noise, not signal
• Corrections that only apply to your specific project — keep skills generic
• Patterns already well-covered by capire — link instead of duplicate

Session learning template

When updating a skill based on session learnings, add a comment in the PR:

## Session learnings — [date]

**Project**: [brief description]
**Model**: claude-sonnet-4 / gpt-4o / etc.

**What triggered this update:**
The agent consistently used X pattern instead of Y when doing Z.

**What changed in the skill:**
Added "❌ X — because [reason]. Use Y instead." to the mistakes section.

**Eval added:**
Eval #[N] — tests that the agent now uses Y when asked to do Z.

This creates a traceable history of why each pattern exists in the skill — which is invaluable when a new model ships and you're deciding what to remove.

---

Skill quality checklist (for PR review)

Before opening a PR for a new or updated skill:

• name and description in frontmatter
• Description is specific enough — tested against 3 real prompts
• Description is not so broad it triggers on unrelated tasks
• Capire doc reference at the top of the body
• All code examples are correct and copy-pasteable
• "Common mistakes to avoid" section with ❌/✅ pairs
• Under 500 lines
• At least 2 evals added to tests/skillOverflow-evals.json
• Baseline comparison done (with vs. without skill)
• Smoke test passes: bash tests/smoke-test.sh .
• If updating: session learnings documented in PR description

Common mistakes to avoid

• ❌ Writing the eval prompt to match the skill wording — biases results, skip the baseline test

• ✅ Write eval prompts as a developer would naturally ask, then check assertions

• ❌ Creating a new skill when an existing one just needs a new section

• ✅ Check ls ~/skillOverflow/skills/ first — update before creating

• ❌ Covering every edge case in one skill — it becomes too long and unfocused

• ✅ One clear, opinionated path per skill — link to capire for the rest

• ❌ Never revisiting skills after the model gets updated

• ✅ Re-run evals on every major model release — remove what the model now knows

• ❌ Keeping skills that produce identical output to the no-skill baseline

• ✅ If baseline comparison shows no difference, the skill adds no value — remove or rewrite it

• ❌ Treating skills as static documentation

• ✅ Every real project session is a source of improvements — ask the agent what it got wrong