Personal (~/.copilot), the current workspace, or Copilot customization files at arbitrary paths: set up a Waza evaluation project, perform static/runtime evaluation, run individual gates, summarize results, and update reports.
Scope
- Default evaluation target:
$HOME\.copilot(instructions / skills / agents) - Example workspace evaluation target:
<workspace>\.github - Example arbitrary evaluation target: a directory or single file containing instructions / agents / skills
- Default Waza evaluation project:
~/waza-copilot-customization-evals - Default static eval:
evals\copilot-customizations\eval.yaml - Default runtime eval:
evals\copilot-customizations\runtime-eval.yaml - Default result JSON:
results\customizations-eval.json - Default runtime result JSON:
results\runtime-customizations-eval.json - Default report:
reports\YYYY-MM-DD-copilot-customizations-waza-report.md
Setup Links
- Setup guide: https://github.com/aktsmm/Agent-Customization/blob/master/docs/waza-copilot-customizations-setup.md
- Template: https://github.com/aktsmm/Agent-Customization/tree/master/templates/waza-copilot-customization-evals
If you copied only this skill, open the setup guide first.evalProject If is missing, start from the template.
Arguments
Skills have no input variable mechanism. Read the following values from the request text and argument-hint , and use the defaults if not specified.
targetScope:personal(~/.copilot) /workspace/custom. Default ispersonaltargetPath:customis required whenworkspaceand empty,<workspace>\.github、personaland empty,$HOME\.copilotevalProject: if empty,~/waza-copilot-customization-evalsaction:setup/run/runtime/gate/report/full. Default isfullexecutionMode:static/runtime/both. Default isstaticgateMode:metadata/agent-structure/token-budget/all。actionがgateused only when . Default isall(logical specification that runs the 3 modes sequentially)maxFileTokens: default is 5000maxTotalTokens: default ispersonalが 130000、workspace/customが 60000runtimePrompt:runtimeprompt to run and record when . If empty, use a smoke prompt that summarizes the targetruntimeModel: model for runtime / copilot-sdk. If empty, check the model names available in the current environment and choose oneruntimeTaskStrategy:template/record. Default istemplate
Operating Rules
- This skill does not modify the instruction / agent / skill bodies being evaluated.
- Make fixes only when the user explicitly says so. For evaluation only, you may update only the eval assets under the Waza project, the result JSON, and the report.
- Do not write secrets, credentials, or personal account values in the report. If there is an issue, write only the file name and the aspect.
- Assume PowerShell; run commands non-interactively, one at a time, with a timeout.
- If the result output is long, extract and report only the summary, failed gates, issues, warnings, and next fix candidates.
action=setuporaction=full: if necessary, you may create or update the.waza.yaml、static/runtime eval、task、grader script、report script、results/、reports/` on the Waza project side.action=reportreads existing result JSON / report first, and if absent, runsrunbefore creating the report.runtimeはcopilot-sdkassumption. If necessary, check thewaza models、copilot loginstate and model name.runtimeIf you want to verify the actual behavior of the target itself with , by default create a deterministic task withwrite-runtime-task.ps1.waza new task from-promptis convenient for exploratory purposes, but it tends to generate graders that are overfitted to temporary paths or specific wording. Do not use it by default, and even when using it, always review the task after generation.runtimePromptis blank, you may auto-generate a smoke prompt that includes the target path and output format.- In the runtime task, do not leave unexpanded variable names (e.g.,
$resolvedTarget,$targetName), and confirm that the resolved target path and basename are present literally. - Before runtime execution, read the task file once and confirm that the target path, expected headings, and grader strictness are as intended.
Workflow
evalProjectis blank, useJoin-Path $HOME 'waza-copilot-customization-evals'.targetScopeとtargetPathDetermine the evaluation target frompersonalor blank:targetPathis empty,$HOME\.copilotworkspace:targetPathis empty, treat<workspace>\.githubcustom:targetPathas required- Either a single file or a directory is acceptable
maxTotalTokensis empty, use 130000 forpersonalwhich includes all personal assets, and use 60000 forworkspace / customwhich assumes the read unit
- Check whether the evaluation target exists. If it does not exist, present candidate paths and stop.
actionDeterminesetup: Only check for the existence of the Waza project and create/update missing assetsrun: Run the static eval and update the result JSON and reportruntime:copilot-sdkrecord and execute the runtime task, and update the result JSON and reportgate: Run only the specified gatereport: Summarize existing results and update the report. Complete execution if necessaryfullor blank: Perform setup -> static run -> optional runtime -> report in order
- Check whether the evaluation project exists.
.waza.yamlevals\copilot-customizations\eval.yamlevals\copilot-customizations\runtime-eval.yamlevals\copilot-customizations\scripts\grade-copilot-customizations.ps1evals\copilot-customizations\scripts\write-copilot-customizations-report.ps1- If it is not found, guide the user to the public repo's setup guide / companion template, and prioritize initializing from the template
action=setuporfull, create or update the missing assets..waza.yamlevals\copilot-customizations\eval.yamlevals\copilot-customizations\runtime-eval.yamlevals\copilot-customizations\tasks\customization-static.yamlevals\copilot-customizations\runtime-tasks\evals\copilot-customizations\scripts\grade-copilot-customizations.ps1evals\copilot-customizations\scripts\write-copilot-customizations-report.ps1results\reports\
- Check whether Waza standalone is available.
waza --versionGet-Command waza
- Place the evaluation target in
$env:WAZA_CUSTOMIZATIONS_TARGET. action=runorfull, run the static eval.waza run evals\copilot-customizations\eval.yaml --output results\customizations-eval.json --no-update-check -v
action=runtime, oraction=fullandexecutionMode=runtime/both, create and execute a runtime task.
- By default
runtimeTaskStrategy=templateuseswrite-runtime-task.ps1use runtimePromptis empty, assemble a smoke prompt that summarizes the target and extracts constraintswrite-runtime-task.ps1 -TargetPath <target> -OutputPath evals\copilot-customizations\runtime-tasks\runtime-target.yaml- After generating the task, read
runtime-target.yamland confirm that the literal target path andPurpose:/Guardrails:/Usage:are included runtimeTaskStrategy=recordonly whenwaza new task from-prompt ... --overwriteuserecordwas used, read the task file and relax the fixed path_pattern or excessively strictcontains_csbefore executionwaza run evals\copilot-customizations\runtime-eval.yaml --output results\runtime-customizations-eval.json --no-update-check -v
action=gatewhengateMode, run only the necessary gates according to
metadataagent-structuretoken-budgetall: grader にMode allwithout passing , execute the above three modes sequentially, one at a time
- Read the result JSON and extract summary.
- total tests
- succeeded / failed / errors
- success rate
- aggregate score
- If there is a failed gate, or when
action=gate, run the necessary individual gates standalone with the same$env:WAZA_CUSTOMIZATIONS_TARGETstill specified to get details. action=reportorrun/runtime/fullafter,write-copilot-customizations-report.ps1usingreports\YYYY-MM-DD-copilot-customizations-waza-report.mdcreate or update.
- Date
- Model: GitHub Copilot
- Action
- Evaluated source
- Commands run
- Waza summary
- Gate results
- Findings
- Next fix candidates
- In the final answer, concisely present the action, execution mode, evaluation target, results, red gates, warnings, and next steps.
Command Reference
Typical static eval. For workspace / custom target, follow the target resolution rules of Workflow and change only $env:WAZA_CUSTOMIZATIONS_TARGET .
$project = Join-Path $HOME 'waza-copilot-customization-evals'
Set-Location $project
$env:WAZA_CUSTOMIZATIONS_TARGET = Join-Path $HOME '.copilot'
waza run evals\copilot-customizations\eval.yaml --output results\customizations-eval.json --no-update-check -v
- For runtime, record, individual gate, and report commands,
Workflowsteps 10–14 are the authoritative source. recordread the generated task before execution and fix fixed temporary paths or overly strictcontains_cs.- For setup / repair, check the required assets from step 5 with
Test-Pathand restore only the missing ones from the template.
Final Response Format
## 結果
- Action: <setup/run/gate/report/full>
- Execution mode: <static/runtime/both>
- Target: <path>
- Waza eval: pass/fail
- Success rate: <value>
- Aggregate score: <value>
- Result JSON: <path>
- Report: <path>
## 赤 Gate
- <gate>: <issue>
## 警告
- <warning>
## 次の一手
1. <action>
2. <action>
赤 Gate や 警告 even if there is noなし explicitly state.