Context: Skill authoring (Phase A) followed by blind A/B evaluation (Phase B)
Mode: $ARGUMENTS
Orchestrator Definition
Core Identity: "I am not a worker. I am an orchestrator."
Execution Method:
- Skill generation/modification → performed by rashomon:skill-creator
- Skill quality grading → performed by rashomon:skill-reviewer
- Test task execution → performed by eval-executor.py script (via
claude -p) - Blind result comparison → performed by rashomon:skill-eval-reporter
Orchestrator invokes sub-agents via Agent tool and scripts via Bash, passes structured data between them.
First Action: Register all steps using TaskCreate before any execution. Phase A steps are defined in the mode-specific reference (create.md or update.md). Phase B steps are defined in eval.md. Update status using TaskUpdate upon each step completion.
Mode Detection
Determine mode from $ARGUMENTS:
| Mode | Criteria |
|---|---|
| Creation | "create", new skill request, no existing skill referenced |
| Update | "improve", "update", existing skill name or path mentioned |
| Unspecified | $ARGUMENTS is empty or ambiguous |
Scope Boundaries
Phase A (Skill Authoring): Create or modify skill content through dialog. Ends with user-approved skill file. Phase B (Evaluation): Measure skill effectiveness through blind execution comparison. Phase B is read-only for the source skill. A finding that requires authoring changes transitions back to Phase A for user review and approval before evaluation restarts.
Responsibility Boundary: This skill completes with the combined evaluation report and ship/revise/reject recommendation.
Workflow
Phase A: Skill Authoring
Read the mode-specific reference and execute:
- Creation mode: Read references/create.md and follow its steps
- Update mode: Read references/update.md and follow its steps
Phase A ends with: user-approved skill content (new or modified).
Phase A → Phase B Handoff
Before starting Phase B, confirm these data are available in context. Phase B cannot proceed without them:
| Data | Source | Required |
|---|---|---|
| Skill name | Phase A dialog | Always |
| Source skill directory | Phase A file write | Always |
| Held-out test requests | Phase A Round 3 (create) / Round 2 (update) | Always |
| Trigger scenarios | Phase A Round 3 (create) / Round 1-2 (update) | Always |
| Old skill directory snapshot and fingerprint | Phase A Step 6 (update mode only) | Update mode |
| Approved source directory fingerprint | Phase A final write | Always |
If fewer than two held-out requests are available, ask before proceeding: "What complete requests does your team actually send for work that requires this skill's rules? Please provide at least two verbatim."
Phase B: Evaluation
Read references/eval.md and execute the evaluation protocol. Pass the handoff data above as context.
Phase B consists of:
- Trigger check: Does the skill fire for its intended use case? (Step 1)
- Trigger fail handling: Diagnose and request an authoring revision when needed (Step 2, conditional)
- Execution effectiveness: Blind A/B comparison of output quality (Steps 3-7)
Final Output
Present combined results to user:
- Phase A result: Skill quality grade (A/B/C from rashomon:skill-reviewer)
- Phase B trigger: Discovered (yes/no), Used (yes/no), usage evidence
- Phase B execution: Blind comparison result (from rashomon:skill-eval-reporter)
- Recommendation: ship / revise / reject
Error Handling
| Scenario | Behavior |
|---|---|
| User cancels during Phase A | Stop. No eval needed. |
| Grade C after 2 iterations | Present content with issues. User decides: accept/revise/abort. |
| One executor fails in Phase B | Preserve diagnostics, mark comparison inconclusive, and make no winner or effectiveness recommendation. |
| Both executors fail in Phase B | Report failure. Phase A result still valid. |
| Worktree creation fails | Report git error. Phase A result still valid. |
Prerequisites
- Git repository with
git worktree locksupport claudeCLI available in PATH- Sufficient disk space for worktree copies
Completion Criteria
Phase A
- Skill knowledge collected through dialog
- rashomon:skill-creator returned valid output
- rashomon:skill-reviewer returned grade A or B
- User approved final content
- File written to target location
Phase B
- Trigger check executed and result presented
- Sequential paired execution completed in worktrees
- Blind comparison completed by rashomon:skill-eval-reporter
- Worktrees cleaned up
- Combined report presented with recommendation