feature-smoke-test

Smoke testing is runtime verification; it is not over once you have written the prompt or read the code. You must actually execute and actually inspect the artifacts.

Determine the test scope

Before starting, infer the feature boundaries from the code changes:

git diff main --name-only
git diff main --stat

Based on the set of changed files, explicitly write down:

• The feature boundaries under test
• User-perceivable behavior changes
• Pre-run setup and post-run cleanup
• Evidence that can prove success or failure

If the feature involves state, asynchrony, approval flows, or timing-sensitive logic, assume by default that a single prompt is not enough.

Read the sources of truth first

Read the minimal set of files that define the feature's actual behavior:

• User-facing docs, changelog, or design notes
• The agent prompt or tool prompt that exposes the feature
• The implementation-layer entry point
• Existing tests—when tests describe behavior more accurately than docs, look at the tests first

Do not trust outdated prompt examples. Reconstruct the real tool interface from the code or current docs first.

Formulate a minimal test plan

Cover three scenarios by default:

1. Happy path
2. Boundary conditions, invalid input, or capacity limits
3. Interruption, retry, cleanup, or recovery

For each scenario, record:

• 目标
• 前置准备
• prompt 策略
• 成功信号
• 失败信号
• 需要检查的产物

If the feature has race conditions, you must explicitly annotate the timing. Use long-running commands or deliberate waits to create timing windows; do not use vague phrasing like "just run it again quickly."

Prefer multi-turn prompts

Use a multi-turn flow by default:

1. Exploration turn: have the agent restate the current real interface after reading the sources of truth
2. Execution turn: execute only one scenario
3. Observation turn: only read the output, tool state, and artifact files; do not expand the scope
4. Cleanup turn: stop, roll back, or close stateful resources

Use a single-turn prompt only for simple stateless features.

For multi-turn testing, invoke the CLI separately for each turn, inspect the previous turn's output and artifacts between turns, and then decide the next turn's prompt. Do not stuff multiple turns of prompts into stdin all at once—that is blind writing, and you cannot adjust based on the previous turn's results.

When testing, explicitly require the agent to first list the currently available tools; do not fabricate historical tool names. For a reusable prompt template, see references/prompt-patterns.md。

Run the CLI in isolation in non-interactive mode

Use /tmp as a one-off directory under --work-dirto achieve session isolation. The CLI's session path is determined by ~/.kimi/sessions/<md5(work_dir)>/ ; different work-dirs automatically produce independent session namespaces and will not pollute the sessions of normal projects. Authentication state is kept under ~/.kimi , with no extra configuration needed.

Environment preparation

SMOKE_DIR="$(mktemp -d /tmp/kimi-smoke-XXXXXX)"

If the feature needs repository context (reading code, git information, etc.), copy or symlink the repository files to SMOKE_DIR. If the feature edits files, never use an active repository as the work-dir.

Default execution method

Use this mode for the vast majority of scenarios:

uv run python -m kimi_cli.cli \
  --print \
  --prompt "你的测试 prompt" \
  --work-dir "$SMOKE_DIR"
echo "exit_code=$?"

--print will automatically enable --yolo(auto-approve all actions), suitable for unattended smoke tests.

After executionyou must first check the exit code: a non-zero value means the CLI itself crashed or timed out; you should investigate process-level errors first, then look at the session artifacts.

Alternative execution modes

When the default approach does not meet your needs, choose as needed:

• Long prompts: pass via stdin——cat <<'PROMPT' | uv run python -m kimi_cli.cli --print --input-format text --work-dir "$SMOKE_DIR"
• Structured output: add --output-format stream-json, outputs line-by-line JSON, convenient for programmatic parsing
• Only view the final result: use --quiet, equivalent to --print --output-format text --final-message-only

Notes

• During execution, record these paths:SMOKE_DIR, the final session directory (which can be located via inspect_session.py ), and feature-specific output files.

Deliberate execution

During execution, maintain a brief run log:

• The complete prompt used
• Key tool calls or commands
• Task ids, output paths, or approval ids produced by the feature
• Record timestamps when timing is sensitive

Do not infer correctness solely from the assistant's final text. Runtime files and tool results are the source of truth.

Inspect session artifacts

First check:

• ~/.kimi/sessions/.../context.jsonl
• ~/.kimi/sessions/.../wire.jsonl
• Session-level files created by the feature

Use scripts/inspect_session.py to find and summarize the latest session:

uv run python .agents/skills/feature-smoke-test/scripts/inspect_session.py --share-dir ~/.kimi

Script exit code meanings: 0 = normal summary, 1 = session directory missing or unparsable.

If the feature creates background tasks, notifications, or ancillary files, inspect these files directly; do not rely solely on model summaries.

Report conclusions

Divide results into three categories:

• Confirmedbehavior
• Behavior that does not matchexpectations
• Still ambiguous, behavior that needs more deterministic reproduction

For each bug or regression, record:

• The prompt that triggers it
• The exact session path
• The artifact paths that prove it
• The minimal reproduction steps that can be derived

Issue investigation

When you find behavior that does not match expectations, do not stop at the reporting level. Launch parallel investigation processes to locate the root cause:

Investigation strategy

For each discovered issue,launch multiple independent investigation directions simultaneously(use the Agent tool to execute in parallel). Based on the specific manifestation of the issue, judge for yourself the most valuable investigation angles; common directions include but are not limited to:

• Trace the actual execution path along the call chain from the entry point that triggers the issue
• Check how the input data changes after passing through each processing stage
• Check whether persistent state (session files, background tasks, notification records, etc.) is consistent
• Run the relevant unit tests and integration tests to confirm whether the tests cover the problematic path

You do not need to mechanically cover all directions. Based on the specific anomaly signals in the session artifacts, choose the 2-3 directions most likely to hit the root cause and pursue them in parallel.

Investigation output

Report each investigation direction independently:

• The specific direction and scope of the investigation
• The facts discovered (with file paths and line numbers)
• Conclusion for this direction: root cause identified / ruled out / further investigation needed

Comprehensive localization

After aggregating all investigation results, output:

• Root cause: a one-sentence summary of the essential cause of the problem
• Evidence chain: the complete chain from the triggering prompt → code path → point of error → artifact manifestation
• Fix suggestion: the minimal-change solution, with specific files and line numbers
• Regression risk: related functionality that requires additional verification after the fix