A procedure for asking a subagent to perform a task and verifying whether it can complete the workflow relying only on the skill's procedure document. It imitates a user test: give only the goal, observe the work, and after it finishes, ask for the basis of its decisions. Even if the AI agent that implemented the skill uses it itself, that only confirms the plumbing is connected. Because it holds the full context of the implementation, it will not notice deficiencies in the procedure document and will overcome them.
Which skills can be test subjects
A procedure-type skill is used as the test subject as-is. A guideline-type skill has no definition of completion on its own. Give, as the task, work on which that guideline acts, and check whether the guideline was read in and whether its statements are testified to as the basis for decisions, through a post-work interview and transcript audit.
When to run it
Do not run it automatically on every change. When you judge it is worth running, propose it to the user and begin only after obtaining agreement. What is worth it is a change that affects the skill's workflow structure or judgment criteria. Adding or rearranging steps, incorporating new commands, changing the judgment criteria of guidelines, and the like fall under this. Do not propose it for wording fixes or refactors that do not change behavior.
How to instruct the subagent
Specify Sonnet as the model
The smarter the model, the more it compensates for deficiencies in the procedure document on its own, so it does not test the procedure document; therefore, in principle use Sonnet as the model. When the skill under test sets a lower bound on the model tier and Sonnet does not meet it, use the model at that lower bound. Start a new subagent that does not inherit the implementation conversation.
Make it use the changed skill
Make it use the skill directory in the worktree under test, not the plugin-version skill the user has installed on the machine. If the skill is being changed together with a CLI etc. in the same repository, include in the prompt an instruction to reinterpret that CLI as the worktree one rather than the installed one. When the change to be tested is in a guideline-type skill, and another skill used for the task reads in that guideline by name, have its reference target also reinterpreted as the worktree one.
Restrict the sources of information
What is permitted is only: what is given in the task, files under the skill directory and the destinations the skill instructs to reference, help built into commands, and the execution results of commands and tools. What is given in the task means the task statement, the fixture to be worked on, and user answers to questions. Help built into commands means the CLI's--helpLike the schema of a tool, this refers to the description of the arguments and return values that a command or tool itself has. Website documentation is not included. This is to avoid encouraging skills designed to make the agent go look at the web.
What is prohibited is investigating the implementation under test, the repository's documentation, history, or past conversation memory from outside the permitted scope. If one reads the implementation code or git history, one can get the right answer even without a procedure manual. The following are explicitly prohibited.
- The repository's source code, docs/, README, .dev/, CLAUDE.md
- Past conversation memory
- Executing git commands. However, if the work itself given to the subagent requires git, the test executor lifts the prohibition
What the prohibition can block is only the routes one reads by oneself. Information that the agent execution environment provides from the start is picked up in the post-work interview.
Do not hint at the target feature in the task statement
Do not write the command name or hints for the feature to be verified in the task statement. What is being verified is whether the procedure manual alone can lead there. Example: when a wiki page is renamed, links to that page from other pages break. If verifying a feature that rewrites the link sources when renaming, request only "Please change the page title to ◯◯," and do not write "so that links from other pages do not break."
Do not let them fill in by guessing
Instruct them to report "I could not understand" rather than filling in by guessing when something cannot be understood even after reading the procedure manual and the help built into the command. This prevents holes in the procedure manual from being hidden by self-completion.
Relay questions to the user
Subagents often have no means to ask the user questions. When the procedure manual reaches a point that calls for asking the user a question, instruct them not to proceed with a default value or guess, but to report the question as is and stop. Once stopped, relay the question to the user, pass the answer to the subagent, and have it resume. If you answer on their behalf, the basis for that judgment becomes your own assumption rather than the user's answer to the question, and it does not serve as evidence that the procedure manual alone was sufficient to complete the task.
Specify the report format
Have them report the following in chronological order. The third declaration serves as a detector for ambiguity in the procedure manual.
- The procedure manual files read and the procedures followed
- The sequence of executed commands and tool calls, and the key points of each result
- Points where they were unsure in their judgment, and points where they were stuck because it was not written in the procedure manual
Test environment
For work that changes real data, use a local development environment or an environment prepared for testing. Which environment and which target to use is decided by asking the user before test execution.
Scenario design
- Design the scenario as an abstract task that a human would likely instruct, and verify whether the correct procedure can be reached from it by reasoning. Things that can be verified by reading the code, such as argument acceptance or error display, are not primary verification targets, but any problems found will be reported
- Not only the normal case, but also verify whether the cautions written in the procedure document actually work, by intentionally creating situations that step on those traps. Example: for a note about names being automatically changed when an object with the same name as the rename destination already exists, prepare an existing object with the same name and then request the rename to verify
- Create multiple scenario proposals and present them to the user before execution so they can choose. This allows humans to compensate for missing or biased perspectives before execution
- Create fixtures such as the target, link source, and collision counterpart in advance, outside the procedure being tested. As with the task statement, be careful with the wording so as not to hint at the target functionality and not to reveal that testing is being performed
Interview after the work
The subagent's context includes information that the agent execution environment provides from the start. Project instructions, the memory index, git status, etc. fall under this. Since it does not involve a read operation, it cannot be prohibited, and it does not remain in the transcript either. When the work is finished, before closing the session, interview to draw out the basis for the judgments. Do this whether it succeeded or failed.
Perspectives to draw out:
- Which information source each judgment was based on. Have them classified into: the task statement, descriptions in the procedure document, help built into commands, execution results of commands or tools, user answers to questions, information provided by the agent execution environment, prior learning knowledge, or assumptions
- What were the unknowns that were filled in with assumptions? The instruction "do not fill in with guesses" only works for places the subagent was aware were guesses, so also pick these up in the post-work interview
- What information was needed but not visible or not usable? This becomes a candidate for descriptions that should be added to the procedure document
- Were there any places where the procedure document, help built into commands, and execution results disagreed? If so, which was adopted? Even if it was overcome, the discrepancy itself becomes a candidate for revising the procedure document
Since answers can be after-the-fact explanations, treat them not as a faithful record of introspection but as testimony for isolating causes, and cross-check them with a transcript audit. If, among the judgments that influenced success or failure, any appear that could not have been reached without one of the following: information provided by the agent execution environment that is not included in the permitted information sources, prior learning knowledge, or assumptions, then that run is not evidence that it could be completed with the procedure document alone. Even for runs where the prohibition was removed, if the shortage in the procedure document was compensated for from branch names or the commit log, then it is not evidence that it could be completed with the procedure document alone.
Not all detected holes must be filled in the procedure document. If even holes that would be stably and correctly compensated are filled, excessive detail will impair the skill's adaptability. Whether the compensation method is stably correct cannot be observed from this test, which forbids inference, so whether to fill each hole is decided by the user on a case-by-case basis. For the judgment, use the kuden:agent-skill skill's question of whether to accept review feedback, and for the points where the subagent got stuck, distinguish whether the model would be stuck without additional description, or whether a smart model could judge on the spot. When filling a hole, the location to modify is determined according to the rules in the subject skill's repository, such as the responsibility boundary between the CLI and the skill, if such rules exist.
Verification of results
Do not take the subagent's self-report at face value; perform two independent checks.
Audit the tool call sequence in the transcript log
Reading the entire transcript would overflow the context, so first extract the sequence of tool calls and read the tool names and main arguments. For calls whose basis for judgment needs to be confirmed, such as the git command in the run where the prohibition was lifted, also read the execution result.
Points to check:
- Whether prohibited information sources were not read
- In runs where the prohibition was not lifted, whether git commands were not executed. In runs where the prohibition was lifted, whether the lack of the procedure document was not compensated from branch names or commit logs
- Whether the built-in help was read before executing the command
- Whether the call sequence matches the self-report
Confirm the actual data of the final state
Read the target directly and confirm whether the intended changes are reflected and whether things that should not be involved remain intact.