SuperAPI AI Directory
Partner中文AI newsFavorites
Skillssetting-up-regression-runs

setting-up-regression-runs

Testing & quality

View repository

Updated 10-31 06:36

About this skill

Setting Up Regression Runs

Bitwarden's manual regression suite for a release is a fixed set of Testmo runs, one per spec in specs/, all linked to the release's milestone. This skill decides which specs a release needs; Skill(bitwarden-testmo-tools:creating-test-runs) creates them. If that skill is not available, stop and ask the user to install bitwarden-testmo-tools.

Release profiles

release-profiles.json lists every spec in each profile:

  • partial — Web (Password Manager, Admin Console, Admin Portal / SM / Providers), Old Client / New Server, Mobile (iOS, Android), and Directory Connector (BWDC).
  • full — the same Web, Old Client / New Server, and Mobile runs, plus CLI, 3 Desktop and 4 Extension configuration variants. No BWDC.

full is not a superset of partial. Use exactly the list in the file.

Workflow

  1. Collect the inputs from the user: the profile (partial or full), the milestone's exact name (e.g. 2026.8.0 Manual Regression), and any spec to skip this release. Ask for whichever is missing.
  2. Confirm the milestone exists. The user creates it in the Testmo UI — the API cannot. Releases use a parent YYYY.M.0 Release milestone with … Manual Regression and … Automated Regression children; manual regression runs link to the … Manual Regression child.
  3. Build the spec list. Read the profile from release-profiles.json and drop any skipped specs. A name the user asks to skip that is not in the profile is a typo — stop and ask rather than continuing. Turn each remaining name into ${CLAUDE_PLUGIN_ROOT}/skills/setting-up-regression-runs/specs/<name>.json, written out as a literal absolute path. Never pass ${CLAUDE_PLUGIN_ROOT} itself to creating-test-runs — in that skill it means the bitwarden-testmo-tools root, not this plugin's.
  4. Hand off. Invoke Skill(bitwarden-testmo-tools:creating-test-runs) and follow it, passing every spec path as its own --spec and the milestone as --milestone-name. That skill owns the dry-run, the user's review, and --create — never run the dry-run and --create in the same turn.
  5. Finish the multi-configuration runs. Every Mobile, Desktop, and Extension run needs a manual Testmo UI pass after creation (full has all three; partial only Mobile). Tell the user which runs still need it, using references/regression-configurations.md and each spec's _comment.

To add one run that is missing from an already-populated milestone, pass only that spec — the script's per-run duplicate check leaves the existing runs alone.

Changing the suite

The specs and profiles here are shipped in the plugin; edits in the installed copy are overwritten by /plugin update. If a spec is skipped every cycle, or a new run joins the suite, change specs/ and release-profiles.json in a PR to bitwarden/ai-plugins.

Case counts drift as the repository changes. Compare each dry-run against the previous cycle; a large swing usually means a renamed folder or tag.