SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillssmoke-regression

smoke-regression

Testing & quality

View repository

Updated 10-16 08:31

About this skill

Smoke / regression pass

The matrix, its fixtures and how to read a failure live in docs/testing/smoke-regression.md. Read §2.2 before running L1 and §3 before L2. This skill is the procedure.

0. Decide the layers

From the diff, pick per §4 of the doc:

  • L0 always, for the packages touched.
  • L1 when the change touches inference_api/chat/*, agents/main_agent/**, apis/shared/harness/**, the SSE table, or any interrupt flow.
  • L2 when the change touches a rendered surface (composer, loader, tool rail, pickers, banners) or an interrupt flow.

Say which layers you are running and why in one line before starting.

1. L0 — hermetic

From the branch's checkout. In a worktree PYTHONPATH is mandatory or you test develop:

cd <checkout>/backend && PYTHONPATH=$PWD/src AWS_EC2_METADATA_DISABLED=true <main checkout>/backend/.venv/bin/python -m pytest tests/ -q

Frontend cd frontend/ai.client && npx ng test --watch=false; infra cd infrastructure && npx tsc --noEmit && npx jest. Never pass dummy AWS credentials to make a stall go away (see the doc §1).

2. L1 — turn matrix

Pick the target in this order and state which one you used:

  1. Deployed dev through the Runtime (always available to an agent with the dev-ai profile and an active headless grant): 
    cd backend && AWS_PROFILE=dev-ai uv run python scripts/smoke_turns.py --auth headless-grant --user-id <sub> --prefix dev-boisestateai-v2 --json /tmp/smoke.json
    The grant's owner id is in dev-boisestateai-v2-bff-sessions under PK=HEADLESS-GRANT#… (grant_user_id). If the script reports no active grant, the user must sign in and run a headless run once; do not try to mint tokens another way (the doc explains why the other routes are dead ends). This mode skips the app-api-only rows (409, Stop, steer). It deletes the sessions it created; after a crash or --keep, pass --delete-sessions id,id.
  2. Local stack when the branch is not deployed yet and you need the app-api rows: app-api on :8000 with SKIP_AUTH=true, inference-api on :8001, then --base-url http://127.0.0.1:8000. Do not flip SKIP_AUTH in the user's backend/src/.env without asking; use a worktree copy.
  3. Deployed through app-api with a Cognito password user (--auth cognito), when one exists for the environment.

Add --with-attachments when attachments are in scope, and the fixture flags (--approval-tool, --skill-id, --agent-id) when those rows matter. Never pass --allow-prod.

Paste the report table into your summary. Treat SKIP as "not covered here", OBSERVED as information, WARN as a model-quality note. A FAIL is a finding: quote its detail, check the three traps in the doc §2.2 before calling it a bug, then fix or report.

3. L2 — in-app browser

Use the built-in browser pane on https://dev.boisestate.ai; it carries the user's session. Install the frame-capture snippet from the doc §3 with javascript_tool first, then work through the checklist rows that apply. Verify with read_page and window.__frames, not screenshots, except for a final visual proof. Mirror the CSRF cookie into X-CSRF-Token for any manual non-GET call. Delete the smoke conversations when done.

4. Report

One table per layer run, a one-line verdict, and the exact commands used so the run is reproducible. If a layer was skipped, say which and why.