SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsvalidate-ui-scenario

validate-ui-scenario

Testing & quality

View repository

Updated 10-16 00:32

About this skill

Validate UI Scenario

Drives a real VS Code instance through a scenario and records reproducible evidence.

Use this to reproduce a reported bug, to show that a fix works, or to attach a recording to a test-plan item. For deterministic regression coverage that runs on every build, write a smoke test instead (see the smoke-tests skill) — this skill is for one-off, issue-derived validation.

A scenario is a small JavaScript file run by test/scenario/out/runScenario.js. Nothing else has to be configured: the runner launches VS Code, records video and a trace, captures a screenshot at every step boundary, writes the report, and captions the recording with each step and its result.

Prepare

npm install                             # once
npm --prefix test/scenario run compile  # after any change under test/scenario

Check ffmpeg and ffprobe are available before running. The runner looks on PATH and in the usual install locations, so an ffmpeg installed after the editor started is still found. Without them the scenario still runs and keeps the raw recording, but the video is not captioned with step titles. The runner warns at startup; if they are missing, tell the user how to install them rather than silently returning an unannotated video:

Platform Install
Windows winget install Gyan.FFmpeg
macOS brew install ffmpeg
Linux sudo apt install ffmpeg

A new terminal may be needed for PATH to pick them up, or set FFMPEG_PATH and FFPROBE_PATH. An existing run can be annotated afterwards with node test/scenario/out/renderEvidenceChapters.js <run-dir>.

Target Flags Also required Use for
Installed Insiders, else Stable (none — the default) nothing Reproducing a report against shipped behavior
Dev build from this checkout --dev Matching Electron runtime, npm run build-fast Verifying an unmerged change
A specific install --build <app-root> nothing Pinning an exact build
Web --web --headless npm run build-fast -- --client-only Browser-only behavior

Reuse the matching Electron runtime when present; run npm run electron only when it is missing or incompatible. Fast builds refresh outputs without type checking; validate types separately when warranted.

With no target flag the runner finds an installed VS Code Insiders (falling back to Stable) and logs which one it chose. --build takes the application root — the install directory on Windows and Linux, or the .app bundle on macOS:

# Windows
--build "C:/Users/<you>/AppData/Local/Programs/Microsoft VS Code Insiders"
# macOS
--build "/Applications/Visual Studio Code - Insiders.app"

Every target runs with its own profile and extensions directory, so your extensions and settings never leak into the recording, and the window is sized to the recording canvas so the capture has no empty margins. The evidence records the quality of the build that actually ran (Insiders, Stable, Dev), so a report always names the product it validated. An installed build only reproduces shipped behavior — to validate an unmerged change, use --dev in a checkout that contains it.

Write the scenario

Save the file next to the run it produces, for example .build/vscode-playwright-mcp/<issue>.cjs. The .cjs extension matters: this package is an ES module package, so a CommonJS scenario named .js fails to load. An ES module scenario with a default export works too.

const os = require('os');
const path = require('path');
const fs = require('fs');

const workspacePath = path.join(os.tmpdir(), 'issue-250159-workspace');
fs.mkdirSync(workspacePath, { recursive: true });

// The settings tree is virtualized, so only the rows near the viewport exist in
// the DOM. Scroll the whole list, otherwise "the setting is absent" cannot be
// told apart from "the setting is below the fold".
const COLLECT_TITLES = `(async () => {
	const editor = document.querySelector('.settings-editor');
	const scrollable = editor.querySelector('.settings-tree-container .monaco-scrollable-element');
	const titles = new Set();
	const collect = () => editor.querySelectorAll('.setting-item-label')
		.forEach(node => titles.add(node.textContent.trim()));
	collect();
	for (let previous = -1; scrollable && scrollable.scrollTop !== previous;) {
		previous = scrollable.scrollTop;
		scrollable.scrollTop = previous + scrollable.clientHeight;
		await new Promise(resolve => setTimeout(resolve, 180));
		collect();
	}
	return [...titles];
})()`;

module.exports = {
	id: 'vscode-250159-settings-search',
	title: 'Settings search matches across title and description',
	source: 'https://github.com/microsoft/vscode/issues/250159',
	workspacePath,
	steps: [
		{
			id: 'SS-01',
			title: 'Open the Settings editor',
			async run(context) {
				await context.workbench.quickaccess.runCommand('workbench.action.openSettings2');
				await context.page.waitForSelector('.settings-editor', { state: 'visible', timeout: 20000 });
				return 'The Settings editor is visible.';
			}
		},
		{
			id: 'SS-02',
			title: 'Search across title and description',
			async run(context) {
				await context.workbench.settingsEditor.searchSettingsUI('chat confirm');
				const titles = await context.page.evaluate(COLLECT_TITLES);
				if (!titles.some(title => /max\s*requests/iu.test(title))) {
					throw new Error(`Max Requests is absent. Found: ${titles.join(', ')}`);
				}
				return 'Max Requests is present in the results.';
			}
		}
	]
};
Field Meaning
id, title Identify the run; id names the evidence directory
source Issue or test-plan item the scenario came from
workspacePath Disposable folder to open
userSettings Settings seeded into the profile before launch, including the built-in Agents profile when using --agents
extraArgs Extra VS Code command-line arguments
extraEnv Environment overrides for the launched app; undefined unsets a variable. Keep the automation runner's HOME unchanged and isolate application discovery with this field.
stepPauseMs How long to hold each finished step so its caption is readable. Defaults to 1000; set 0 when the scenario is timing-sensitive

Each step receives a context with app, workbench, code, page, and skip(reason, options). workbench exposes the feature helpers (settingsEditor, quickaccess, editors, terminal, chat, …); page is the Playwright page for anything they do not cover.

Use context.code.driver.resizeWindow({ width, height }) for responsive-layout scenarios. It resizes the current native Electron window (or the browser viewport on web) and returns the previous size for restoration.

  • Return a string describing how the step was validated. It appears in the report.
  • Throw to fail the step. The message is recorded, and the run stops.
  • Call skip(reason, { needs }) when the step cannot be validated automatically. The run stops and is reported as aborted, never as passed.

Steps that cannot be automated

Decide this while planning, before writing the scenario, and classify each one — the two kinds have different consequences:

needs Meaning What to do
human A person is required: physical hardware, a subjective judgement, a sign-in that cannot be scripted Report the step so someone can check it by hand
infrastructure Automatable in principle, but the harness cannot do it yet Report it as an enhancement to this skill, naming the missing capability 
ctx.skip('Comparing physical print output requires a person with a printer.', { needs: 'human' });
ctx.skip('The harness cannot drive native OS file dialogs.', { needs: 'infrastructure' });

Blocked steps are recorded in manifest.json, highlighted in a Needs attention section of report.html, marked on the video caption (SKIPPED - NEEDS HUMAN), and printed at the end of the run. Surface them in your summary — never quietly drop a step you could not perform, and never weaken an assertion so that it passes.

Run it

node test/scenario/out/runScenario.js <scenario.cjs>

Exit code 0 means every step passed, 1 means the run failed or was aborted, 2 a usage error.

Evidence is written to .build/vscode-playwright-mcp/evidence/<run-id>/:

File Contents
report.html Step table, outcome, embedded video
manifest.json Step timestamps, statuses, artifact paths, environment
videos/annotated.mp4 Recording with a caption band showing each step and its validation result
videos/*.webm The raw recording
*.png Per-step screenshots
logs/ Playwright trace, window and server logs

The caption band is added above the recorded frame rather than drawn over it, so no recorded pixel is hidden and the recording keeps its original length. Each caption carries the step number and id, its status, the step title, and the validation detail the step reported. Re-render after editing a manifest with node test/scenario/out/renderEvidenceChapters.js <run-dir>.

What makes evidence trustworthy

  • Assert on DOM state, accessibility, focus, or text — screenshots support a claim, they do not establish one.
  • Validate through a signal separate from the action. An automation call returning successfully is not a result.
  • Beware virtualized lists. The settings tree and long lists render only the rows near the viewport, so scroll the whole list before concluding that something is absent.
  • If the bug is a race, make the timing explicit — a forced delay or a repeated loop — so the recording shows the window in which it occurs rather than relying on luck.
  • Record the failing behavior before the fix when you can. A passing run alone does not show that the scenario would have caught the bug.

Report back

Summarize the outcome, list failed or skipped steps, link report.html, and state the OS, the VS Code version and quality (both are in manifest.json), and the source issue. Attach the video to the issue or pull request by dragging it into the comment box.

Always call out, separately from the pass/fail result:

  • steps that need a person, so someone knows what is still unverified;
  • steps blocked on a missing harness capability, named as a concrete enhancement to this skill;
  • anything that degraded the evidence, such as a missing ffmpeg leaving the video uncaptioned.

Related

  • Interactive exploration. test/mcp also serves these tools over MCP (vscode_automation_*), which helps when you need to inspect the UI before knowing what to assert. Configure it as an MCP server with cwd test/mcp and command npm run start-stdio.
  • Automated validation on a pull request. microsoft/vscode-engineering runs the same harness in CI: labelling a pull request ~requires-ui-validation researches the change, runs a scenario against the exact merge candidate, and posts the per-step result with captioned video. Use this skill when a scenario is not yet covered there, or to iterate locally before proposing one.
  1. Confirm ffmpeg/ffprobe are available; if not, say so and give the install command before running, so the user is not surprised by a video without step titles.
  2. Read the issue and identify the observable claim: searching chat confirm in the Settings editor should match Max Requests, whose description mentions confirmation.
  3. Add a baseline step (max requests finds the setting) so a failure cannot be explained by the setting being missing from the build.
  4. Write .build/vscode-playwright-mcp/issue-250159.cjs and run it with no target flag, which uses the installed Insiders; read the printed report path.
  5. Report the outcome per step, link report.html, and attach videos/annotated.mp4.

The run fails at the search step, and that is the answer: the issue reproduces. Report it as a successful reproduction, not as a broken scenario.