visual-qa

Skill: visual-qa — visual self-check workflow (capture → diff → judge → fix → re-verify)

Why it exists: an agent edits UI and then self-praises is a systematic bias (failure ui-pass-without-full-visual-review). Screenshots taken but no one opens them = as if not tested. This workflow replaces "agent looks itself then passes itself" with three independent check layers, of which two layers are machines.

Decision 13/07/26 (from /last30days): the industry has only ONE thing that is mature for AI-generated UI — baseline snapshot + pixel diff (argos-ci/argos: "detects unintended visual changes"; Playwright toHaveScreenshot). The layer "LLM-judge scores screenshots against a rubric" is still frontier (Agent-as-a-Judge, paper-stage) — so we do that layer ourselves , not waiting for anyone.

WHAT

Purpose and context

• Purpose: replace "agent looks itself then passes itself" with three independent check layers for a local SPA — baseline pixel-diff (machine), design/a11y invariants (machine), reading all images against a rubric (agent) — and pass only when machine AND eyes are both clean, with image files as evidence.
• Trigger (when to use):
  • Just changed theme / UI / rebrand and need to know whether anything was broken unintentionally.
  • Frame has ui_role≠none needs an acceptance gate (see skill br § Visual self-check loop).
  • Browser extension cannot capture localhost (common) → headless CLI as a substitute.
  • The user says "visual QA", "visual qa", "route-shots", "screenshot each route", "check the interface", "baseline diff", "does the UI have a regression?", "test the interface headlessly", "/visual-qa".
• Non-goals: do not step on computer-use (desktop AX / Orca browser); not a 4-item senior verdict (that is /qc-uiux, use this engine); do not loosen thresholds to "clear errors".

Mental model

app chạy + route list THẬT + login → route-shots (ảnh + MANIFEST, mọi theme/state) → tầng 1 baseline diff (ngưỡng 0) → tầng 2 bất biến in-page (contrast-aa · monochrome-surface · rogue-slab · shadow-clipped · overlap · row-misalign) → tầng 3 đọc đủ N ảnh theo rubric → FINDINGS.md → sửa → chụp lại + đọc lại.

Input and output contract

	Field	Required?	Meaning
In	app is running (--base)	Yes	localhost SPA
In	route list from the REAL router + login account	Yes	do not guess; if login fails the gate must die
In	theme/state list from the app itself	Yes	do not hardcode light,dark when the app has more
In	baseline directory	No	created automatically on first run
Out	<out>/*.png + MANIFEST.json	Yes	no file = NEVER CAPTURED
Out	exit code with --assert	available for gate	1 when route errors / missing images / invariant violation / baseline drift
Out	FINDINGS.md	Yes	one line for EACH route, including "clean"

Rules and capabilities

Rules drawn from blood — don't repeat:

• RULE-01 (MUST): ANYTHING COMPUTABLE MUST be computed by MACHINE — forbidden to delegate to the eye. Text contrast is a number (WCAG ratio = f(text color, background color)). Rubric says "text meets AA" but no invariant installed ⇒ evaluator BLIND: faded text slips through the entire login page, and the agent looking at the image can't imagine it either. Any quantifiable criterion → must become an assert, not a line of text in the rubric.
• RULE-02 (MUST): NO ROUTE MAY BE A BLIND SPOT. Driver login-first ⇒ /auth redirect ⇒ the sign-in page is never audited — exactly the page with the bug. Must capture the NOT-LOGGED-IN state first, then log in. Recheck: what state (modal, error, empty, loading) has never entered the frame?
• RULE-03 (MUST): THEME IS A STATE — capture BOTH light AND dark. Apps usually enable dark via a class (.dark on <html>) according to app setting, NOT according to prefers-color-scheme ⇒ headless by default only ever sees light, while dark is an absolute blind spot: a user opening the app in dark sees broken UI while the gate stays green. Enable --themes light,dark (default) → first run reveals 172 places failing AAA in dark while light is spotless. Design consequence: accent must FLIP with the theme (light surface → dark accent; dark surface → light accent); a single accent constant for both themes is wrong by definition. And forbidden to hardcode text color on accent (color:#fff) — in the other theme it becomes white-on-white 1.13:1.
• RULE-04 (MUST): COVER EVERY THEME THE APP PROVIDES — the list comes from the APP, not made up. I hardcoded light,dark while the app has three themes (default / default-dark / paper) ⇒ paper nobody touched it ⇒ it breaks exactly like dark, users find it, not the gate. Rule: before running, read the theme list from the app itself (Settings→Preferences dropdown / source THEME_OPTIONS) then cover all of them. By extension: every STATE AXIS (theme, locale, role, empty/full data) must come from the app, not from the agent's memory.
• RULE-05 (MUST): CHANGING STATE MUST GO THROUGH THE RIGHT APP PATH — then VERIFY it actually changed. Wrong at 2 consecutive layers: ① attaching the class .dark by hand (the app uses data-theme + inject CSS ⇒ the mechanism DOES NOT EXIST); ② setting localStorage (the app OVERWRITES it back from the account setting on the server). Both times the gate went GREEN on a state that is not real — worse than red. The certain right path is the path the user clicks (Settings → Preferences → select). And there is always the invariant: two different themes with identical backgrounds ⇒ FAKE image ⇒ ASSERT FAIL (the gate is screenshotting the same theme twice and then praising itself).
• RULE-06 (MUST): THEME IS AN APP MECHANISM, READ THE SOURCE FIRST. Don't assume .dark / prefers-color-scheme. Grep documentElement.classList / data-theme / localStorage / inject <style> to know what the app uses to change theme; themes are usually loaded via runtime-injected CSS ⇒ it WINS over our static theme file on ordinary tokens, but LOSES to the rule !important ⇒ producing an interface that is neither one thing nor the other (app theme background + our theme card = strange color patches across the page). The correct fix: write our own color palette into the very theme file the app injects, don't write a .dark{} block where the app never reads.
• RULE-07 (MUST): BROKEN LOGIN = THE GATE MUST DIE. Login fail ⇒ every route redirects /auth ⇒ the audit driver 8 copies of the sign-in page and then prints "design-ok". That kind of green is worse than red: it confirms an app it has never seen. --assert must fail the moment login fails.
• RULE-08 (MUST): USER REPORTS A BUG + MACHINE REPORTS CLEAN ⇒ THE MACHINE IS WRONG, NOT THE USER. (lesson 14/07/26 — user: "this is developing, not a courtroom, so why defend your argument so stubbornly".) A green gate is NOT a shield: it only proves what it was taught, on the state it managed to capture. The user seeing a bug = there exists a state/theme/bundle the gate has never entered. It is forbidden to rebut the user with speculation ("probably cache", "probably pressed the wrong key") — every guess pushes the burden of verification onto the user, and I have been wrong 3 times in a row like that. The right process: (1) believe the report, (2) reproduce the user's EXACT state (route + theme + bundle + login/focus state), (3) only speak once you have MEASURED. If you cannot reproduce it, say plainly "not reproducible yet", don't build a hypothesis and then defend it.
• RULE-09 (MUST): REMOVE RANDOMNESS BEFORE SCREENSHOTTING, DON'T LOOSEN THE THRESHOLD. The blinking cursor + animation make every route deviates from baseline by ~0.04% per capture ⇒ gate stays red forever ⇒ gets ignored ⇒ real regression slips through. Driver injects CSS animation/transition:none; caret-color:transparent + hides the CodeMirror cursor right at init ⇒ diff returns to 0%. Loosening --diff-threshold to "stop complaining" is blinding the machine, not fixing the bug.
• RULE-10 (MUST): GREEDY SELECTOR = SHOOTING YOURSELF IN THE FOOT. [class*="cursor"] in the de-noising CSS also hits cursor-pointer/cursor-text of Tailwind ⇒ hides the input field ⇒ login dies ⇒ the gate itself breaks the app it is checking. Any CSS/selector injected by the gate must be as narrow as possible and must be run through the full flow (login → every route) after being added.
• RULE-11 (MUST): THE GATE MUST RUN FROM A CLEAN REPO. Missing playwright-core ⇒ node route-shots.mjs dies right at the import line ⇒ the acceptance_test of the UI frame never runs = the gate is dead and nobody knows. Deps declared in skills/visual-qa/assets/package.json; install: npm i --prefix skills/visual-qa/assets.
• RULE-12 (MUST): EVIDENCE > WORDS. A pass/fixed is only valid when there is an IMAGE FILE. No file = not done. Declaring a pass by inference is forbidden.
• RULE-13 (MUST): READ EVERYTHING, no cherry-picking. Reading 3/7 images and then concluding that is the root cause of the failure ui-pass-without-full-visual-review — the bug is already sitting in the unopened images.
• RULE-14 (MUST): "Machine OK" ≠ "UI beautiful". An invariant only knows what it was taught. A green gate is NOT an excuse to skip the image-reading step.
• RULE-15 (MUST): THRESHOLD is a double-edged sword. Loose threshold → real bugs slip through (a 0.5% diff lets a regression that flattens the whole card slip through). Too tight a threshold → false alarms → noisy gate = DEAD gate. The baseline uses threshold 0 + manual review; DOM invariants must have a threshold that distinguishes real bugs from noise.
• RULE-16 (MUST): CSS must be MEASURED, not GUESSED. Diagnosing by eye was wrong 4 times in a row (clip → collision → wrong selector → wrong specificity). Only when querying getComputedStyle did the root show up. Before every CSS fix: measure computed style + measure rect, then fix. After fixing: measure again before looking at the image.
• RULE-17 (MUST): Don't BLIND the machine to "stop the errors". Lowering blur below the detector threshold = hiding the bug, not fixing it (the aura still gets cropped). Fix the defect, don't fix the measuring stick.
• RULE-18 (MUST): In a container overflow:hidden (narrow rail, tab bar): do NOT use outset shadows. It will get clipped or spill into the gap → ugly streak. Use inset (cannot overflow) or drop the shadow. overflow-x:visible NOT feasible when the other axis is auto (CSS forces it back to auto).
• RULE-19 (MUST): playwright-core + system Chrome (do not download chromium 130MB). Baseline diff needs pixelmatch + pngjs (small, if absent skip baseline automatically).
• RULE-20 (MUST): networkidle is forbidden with realtime SPA — use domcontentloaded + fixed wait.
• RULE-21 (MUST): Do not step on computer-use (desktop AX / Orca browser). This skill is a headless CLI for localhost SPA.
• Capabilities: run headless browser on localhost; measure computed style + in-page rect; compare pixels with baseline; write images/manifest/findings; fix target project UI.

Failure boundaries

• Login fail → --assert failed immediately, do not audit a copy of the sign-in page.
• Missing playwright-core → gate dies from the import line: blocked, install npm i --prefix skills/visual-qa/assets.
• Baseline mismatch ≠ 0 → failed until the viewer + approves with --update-baseline.
• User reports a bug but the machine reports clean → the machine is wrong: reproduce the exact state; if it cannot be reproduced, say plainly "could not reproduce".
• Machine clean but has not read all N images → not done.

HOW

Main workflow

W01–W07 correspond to steps 0–6 below.

Step	Type	Inputs	Action	Outputs/exit	Failure/next
W01	deterministic	app, router, account	Confirm the app runs, get the REAL route list + theme list from the app	route + theme list	app not running → blocked
W02	effect	route list	Capture with route-shots.mjs (+ --assert)	images + MANIFEST.json	login fail → failed
W03	deterministic	images + baseline	Baseline diff threshold 0	diff	≠ 0 → view + approve (B02) or fix
W04	deterministic	in-page page	Machine-checkable invariants	violation or clean	HARD FAIL → W06
W05	judgment	MANIFEST N routes	Read all N images, score by rubric	row for each route	missing unread images → not done
W06	effect	findings	FINDINGS.md (ROOT cause) → fix	code fixed	—
W07	deterministic	code fixed	Re-capture → re-read	clean machine AND clean eyes	still failing → W06

Step-by-step details (source of truth for W01–W07):

0. The app must be running. Know the route list (obtained from the REAL router, not guessed) + login account.

1. CAPTURE + EVIDENCE.

node skills/visual-qa/assets/route-shots.mjs \
  --base http://localhost:5230 --out <dir>/qa-shots \
  --baseline <dir>/qa-baseline \
  --user demo --pass demo --assert

Generate <out>/*.png + MANIFEST.json (route → file → bytes). No file = NEVER CAPTURED.

2. BASELINE DIFF (tier-1 machine) — catch UNINTENDED changes. First time auto-create baseline. Next time compare pixel-by-pixel. Default threshold = 0: every diff ≠ 0 is FAIL and must be viewed + approved manually via --update-baseline. This is the only tier that catches the kind of bug that both CSS-invariants and the agent's eyes are BLIND to (e.g.: over-correcting that flattens the whole UI).

3. MACHINE-CHECKED INVARIANTS (tier-2 machine) — design conformance + a11y. Run in-page using computed style:

• contrast-aa — WCAG AAA for EVERY text node (compute contrast ratio from text color + effective background; 7:1 normal, 4.5:1 large text). Hard FAIL.
• monochrome-surface — large pane (sidebar/aside/nav) must have the same background color. Hard FAIL.
• rogue-slab — EVERY element ≥2% viewport with an opaque background differing from body background > 1.6:1. Hard FAIL. (monochrome-surface only inspects sidebar/aside/nav ⇒ blind to odd color patches that are <div> anywhere mid-page.)
• shadow-clipped — aura outset within parent overflow:hidden (with blur threshold). Warning.
• overlap — two controls (button/a/input/[role=button]) are NOT parent-child but overlap rect ≥4px in both dimensions. Catches the bug buttons overlapping each other (e.g. position:fixed with magic-offset when the badge width varies). Hard FAIL.
• row-misalign — controls with the same parent, roughly the same row (top offset ≤6px) but height difference >3px. Catches the bug adjacent buttons not equal in height (e.g. a toggle button taller than the badge due to different padding/font). Hard FAIL. (These two rules were added 15/07/26 because both /redesign-existing-projects [reads STATIC CSS — cannot measure rects] and unit-tests [check CONTENT, not coordinates] are blind to geometry bugs at render time. The overlap threshold INTENTIONALLY does not use %-of-area: the real bug only clipped the button edge = 18% area, every % threshold would slip through — 2 independent controls overlapping by ≥ a few px is already a bug.)

Extend invariants per the project's design spec.

4. READ IMAGES — HARD COVERAGE + RUBRIC (human/agent tier). MANIFEST has N routes → MUST Read have all N images. FINDINGS must have one line for EACH route (including "clean"). Score by rubric, not gut feeling — e.g. neumorphism: ① single-color surface ② shadows not clipped ③ flat accent CTA ④ text/icons AA ⑤ empty-state on brand ⑥ focus ring.

5. FINDINGS.md → FIX. Table # · route · vấn đề · mức · **nguyên nhân GỐC** · cách sửa.

6. RE-CAPTURE → RE-READ → then pass. Clean machine AND clean eyes. Missing either one = not done.

Branches

ID	Kind	Guard	Behavior	Skip / failure	Rejoin
B01	conditional_required	frame gate only needs 1 route	run --route <r>	—	W03
B02	user_optional	deviation from baseline is an INTENTIONAL change that has been reviewed	--update-baseline	images not reviewed → cannot approve	W04
B03	capability_optional	has pixelmatch + pngjs	run baseline diff	does not have → automatically skip baseline	W04
B04	recovery	user reports error but gate is green	reproduce the EXACT state (route + theme + bundle + login/focus), measure before speaking	cannot reproduce → say so directly	W02

Validation and stopping

Machine: exit code --assert (route error / missing images / invariant / baseline). Eyes: FINDINGS has enough one line for each route in MANIFEST. Pass only when both are clean AND there are image files; no files = not done.

Examples

• Positive: change app theme /br generated, 7 routes × 3 themes from THEME_OPTIONS → route-shots.mjs --assert exit 0, baseline diff 0%, agent Read enough images, FINDINGS 7 lines "clean" → pass with images.
• Boundary/failure: light is clean but dark not yet covered → enable --themes light,dark exposes 172 AAA failures → --assert exit 1; fix accent to flip with theme then re-capture, do not loosen --diff-threshold.

Reference — Driver flags

--base --out --user --pass · --route <r> (1 route, for frame gate) · --assert (exit 1 when: route error / missing images / invariant violation / baseline deviation) · --baseline <dir> · --update-baseline (approve intentional changes) · --diff-threshold <%> (default 0).