SuperAPI AI Directory

evaluate-waza-copilot-customizations

Coding & development

View repository

Updated 11-11 22:14

About this skill

WAZA evaluate copilot customizations

Personal (~/.copilot), the current workspace, or Copilot customization files at arbitrary paths: set up a Waza evaluation project, perform static/runtime evaluation, run individual gates, summarize results, and update reports.

Scope

  • Default evaluation target: $HOME\.copilot(instructions / skills / agents)
  • Example workspace evaluation target: <workspace>\.github
  • Example arbitrary evaluation target: a directory or single file containing instructions / agents / skills
  • Default Waza evaluation project: ~/waza-copilot-customization-evals
  • Default static eval: evals\copilot-customizations\eval.yaml
  • Default runtime eval: evals\copilot-customizations\runtime-eval.yaml
  • Default result JSON: results\customizations-eval.json
  • Default runtime result JSON: results\runtime-customizations-eval.json
  • Default report: reports\YYYY-MM-DD-copilot-customizations-waza-report.md

Setup Links

If you copied only this skill, open the setup guide first.evalProject If is missing, start from the template.

Arguments

Skills have no input variable mechanism. Read the following values from the request text and argument-hint , and use the defaults if not specified.

  • targetScope: personal(~/.copilot) / workspace / custom. Default is personal
  • targetPath: custom is required whenworkspace and empty, <workspace>\.github、personal and empty, $HOME\.copilot
  • evalProject: if empty, ~/waza-copilot-customization-evals
  • action: setup / run / runtime / gate / report / full. Default is full
  • executionMode: static / runtime / both. Default is static
  • gateMode: metadata / agent-structure / token-budget / all。action が gate used only when . Default is all(logical specification that runs the 3 modes sequentially)
  • maxFileTokens: default is 5000
  • maxTotalTokens: default is personal が 130000、workspace / custom が 60000
  • runtimePrompt: runtime prompt to run and record when . If empty, use a smoke prompt that summarizes the target
  • runtimeModel: model for runtime / copilot-sdk. If empty, check the model names available in the current environment and choose one
  • runtimeTaskStrategy: template / record. Default is template

Operating Rules

  • This skill does not modify the instruction / agent / skill bodies being evaluated.
  • Make fixes only when the user explicitly says so. For evaluation only, you may update only the eval assets under the Waza project, the result JSON, and the report.
  • Do not write secrets, credentials, or personal account values in the report. If there is an issue, write only the file name and the aspect.
  • Assume PowerShell; run commands non-interactively, one at a time, with a timeout.
  • If the result output is long, extract and report only the summary, failed gates, issues, warnings, and next fix candidates.
  • action=setup or action=full : if necessary, you may create or update the .waza.yaml、static/runtime eval、task、grader script、report script、results/、reports/` on the Waza project side.
  • action=report reads existing result JSON / report first, and if absent, runs run before creating the report.
  • runtime は copilot-sdk assumption. If necessary, check the waza models、copilot login state and model name.
  • runtime If you want to verify the actual behavior of the target itself with , by default create a deterministic task with write-runtime-task.ps1 .
  • waza new task from-prompt is convenient for exploratory purposes, but it tends to generate graders that are overfitted to temporary paths or specific wording. Do not use it by default, and even when using it, always review the task after generation.
  • runtimePrompt is blank, you may auto-generate a smoke prompt that includes the target path and output format.
  • In the runtime task, do not leave unexpanded variable names (e.g., $resolvedTarget, $targetName), and confirm that the resolved target path and basename are present literally.
  • Before runtime execution, read the task file once and confirm that the target path, expected headings, and grader strictness are as intended.

Workflow

  1. evalProject is blank, use Join-Path $HOME 'waza-copilot-customization-evals' .
  2. targetScope と targetPath Determine the evaluation target from
    • personal or blank: targetPath is empty, $HOME\.copilot
    • workspace: targetPath is empty, treat <workspace>\.github
    • custom: targetPath as required
    • Either a single file or a directory is acceptable
    • maxTotalTokens is empty, use 130000 for personal which includes all personal assets, and use 60000 for workspace / custom which assumes the read unit
  3. Check whether the evaluation target exists. If it does not exist, present candidate paths and stop.
  4. action Determine
    • setup: Only check for the existence of the Waza project and create/update missing assets
    • run: Run the static eval and update the result JSON and report
    • runtime: copilot-sdk record and execute the runtime task, and update the result JSON and report
    • gate: Run only the specified gate
    • report: Summarize existing results and update the report. Complete execution if necessary
    • full or blank: Perform setup -> static run -> optional runtime -> report in order
  5. Check whether the evaluation project exists.
    • .waza.yaml
    • evals\copilot-customizations\eval.yaml
    • evals\copilot-customizations\runtime-eval.yaml
    • evals\copilot-customizations\scripts\grade-copilot-customizations.ps1
    • evals\copilot-customizations\scripts\write-copilot-customizations-report.ps1
    • If it is not found, guide the user to the public repo's setup guide / companion template, and prioritize initializing from the template
  6. action=setup or full , create or update the missing assets.
    • .waza.yaml
    • evals\copilot-customizations\eval.yaml
    • evals\copilot-customizations\runtime-eval.yaml
    • evals\copilot-customizations\tasks\customization-static.yaml
    • evals\copilot-customizations\runtime-tasks\
    • evals\copilot-customizations\scripts\grade-copilot-customizations.ps1
    • evals\copilot-customizations\scripts\write-copilot-customizations-report.ps1
    • results\
    • reports\
  7. Check whether Waza standalone is available.
    • waza --version
    • Get-Command waza
  8. Place the evaluation target in $env:WAZA_CUSTOMIZATIONS_TARGET .
  9. action=run or full , run the static eval.
    • waza run evals\copilot-customizations\eval.yaml --output results\customizations-eval.json --no-update-check -v
  10. action=runtime, or action=full and executionMode=runtime/both , create and execute a runtime task.
  • By default runtimeTaskStrategy=template uses write-runtime-task.ps1 use
  • runtimePrompt is empty, assemble a smoke prompt that summarizes the target and extracts constraints
  • write-runtime-task.ps1 -TargetPath <target> -OutputPath evals\copilot-customizations\runtime-tasks\runtime-target.yaml
  • After generating the task, readruntime-target.yaml and confirm that the literal target path and Purpose:/Guardrails:/Usage: are included
  • runtimeTaskStrategy=record only when waza new task from-prompt ... --overwrite use
  • record was used, read the task file and relax the fixed path_pattern or excessively strict contains_cs before execution
  • waza run evals\copilot-customizations\runtime-eval.yaml --output results\runtime-customizations-eval.json --no-update-check -v
  1. action=gate when gateMode , run only the necessary gates according to
  • metadata
  • agent-structure
  • token-budget
  • all: grader に Mode all without passing , execute the above three modes sequentially, one at a time
  1. Read the result JSON and extract summary.
  • total tests
  • succeeded / failed / errors
  • success rate
  • aggregate score
  1. If there is a failed gate, or when action=gate , run the necessary individual gates standalone with the same $env:WAZA_CUSTOMIZATIONS_TARGET still specified to get details.
  2. action=report or run/runtime/full after,write-copilot-customizations-report.ps1 using reports\YYYY-MM-DD-copilot-customizations-waza-report.md create or update.
  • Date
  • Model: GitHub Copilot
  • Action
  • Evaluated source
  • Commands run
  • Waza summary
  • Gate results
  • Findings
  • Next fix candidates
  1. In the final answer, concisely present the action, execution mode, evaluation target, results, red gates, warnings, and next steps.

Command Reference

Typical static eval. For workspace / custom target, follow the target resolution rules of Workflow and change only $env:WAZA_CUSTOMIZATIONS_TARGET .

$project = Join-Path $HOME 'waza-copilot-customization-evals'
Set-Location $project
$env:WAZA_CUSTOMIZATIONS_TARGET = Join-Path $HOME '.copilot'
waza run evals\copilot-customizations\eval.yaml --output results\customizations-eval.json --no-update-check -v
  • For runtime, record, individual gate, and report commands, Workflow steps 10–14 are the authoritative source.
  • record read the generated task before execution and fix fixed temporary paths or overly strict contains_cs .
  • For setup / repair, check the required assets from step 5 with Test-Path and restore only the missing ones from the template.

Final Response Format

## 結果
- Action: <setup/run/gate/report/full>
- Execution mode: <static/runtime/both>
- Target: <path>
- Waza eval: pass/fail
- Success rate: <value>
- Aggregate score: <value>
- Result JSON: <path>
- Report: <path>

## 赤 Gate
- <gate>: <issue>

## 警告
- <warning>

## 次の一手
1. <action>
2. <action>

赤 Gate や 警告 even if there is noなし explicitly state.