bug-screenshot-annotation

Bug Screenshot Annotation

A workflow skill (metadata.kind: workflow) with no dispatch of its own: it is loaded INSIDE the subagent that is already executing a testing stage, spawns no agents, and is not a stage owner. Typical caller: /sprint-testing Execution, when a bug found during exploration is visual/positional (see ../sprint-testing/references/exploration-patterns.md §"Bugs found during exploration").

Compact Rules

DO NOT route a QA screenshot through ANY external image service, generative or otherwise — real product/customer data is in the frame. Explicit user authorization in chat does NOT lift this; everything renders locally over a loopback HTTP server and a local browser capture.
WHEN a bug is visual or positional (overlap, misalignment, wrong date/offset on an axis, an element in the wrong place) and a raw screenshot would need a paragraph to explain: annotate it. DO NOT use this skill to file the bug itself, or on before/after shots that already read clearly raw.
DO: work from a screenshot that already exists on disk. This skill overlays shapes on an existing image; it never generates or edits an image from a text description.
DO: produce exactly ONE evidence file — the final annotated PNG in the ticket's evidence/ folder, named {KEY}-BUG-{BUG-KEY}-annotated.png. The crop and the annotation HTML are scratchpad working files, never written to evidence/ and never cited from a ticket.
DO: copy the commented overlay blocks from references/shapes.html instead of designing from scratch, and keep both its z-index scale (base image → shapes → callout boxes → corner badge topmost) and its utf-8 meta tag; a copied block that drops either produces a hidden badge or mojibake.
DO NOT: load the annotation HTML over file:// — the browser-automation CLI refuses it before rendering. Serve over loopback HTTP, and kill that server before the session ends.
DO: size the capture viewport equal to or larger than the HTML canvas. A smaller viewport clips callouts.
DO: read the rendered PNG back and expect at least one adjustment pass (move a circle, rewrap callout text, nudge the badge out of a collision). It is not one-shot.
DO: state the final PNG's repo-relative path in chat the moment it lands, unprompted, and repeat it leading the "Bug annotations" group in the session-close screenshot list.
WHEN embedding the annotated PNG into the bug issue: offer it and let the human confirm first; once published it leads the bug's Evidence section, ahead of the raw capture.

Read full SKILL.md when: building the annotation HTML, choosing shape types, or handling a case the local render cannot cover (e.g. a photo of physical signage that would need anonymization).

Why 100% local — the security rationale (binding)

QA screenshots carry real product, customer and competitor data. A screenshot sent to an external image service has already landed in that service's public bucket, and the agent runtime's data-exfiltration classifier blocks such uploads even when the user authorizes them in chat.

So: QA evidence containing real product or customer data never routes through an external image service, generative or otherwise. This skill sidesteps the risk entirely — everything happens with HTML+CSS rendered by an HTTP server bound to 127.0.0.1, captured by a local browser-automation CLI; nothing leaves the machine. If a genuinely unhandleable case appears (e.g. annotating a photo of physical signage), go back to the user and talk through anonymization first — NEVER quietly pipe a real screenshot to an external service.

Scope

Input: a raw screenshot that already exists on disk (typically in the ticket's PBI evidence/ folder). This skill overlays shapes; it does not generate or edit images from a text description.
Output: exactly ONE file that counts as evidence — the final rendered annotated PNG, in the ticket's evidence/ folder. The crop and the annotation HTML are Bucket C working files (see ../agentic-qa-core/references/evidence-conventions.md §1): session scratchpad only, never referenced from Jira/ATR/bug tickets.
Not for: filing the bug (reporting-templates owns that), plain before/after shots that read clearly raw, photos of physical objects/documents.

Process (8 steps + optional embed)

Identify the region. From the raw screenshot (or its accessibility snapshot), work out the pixel crop box and roughly where each annotation shape lands relative to it.
Crop with Python + PIL — scratchpad only.
```
from PIL import Image
img = Image.open("<raw-screenshot-path>")
img.crop((left, top, right, bottom)).save("<scratchpad>/crop.png")
```
The crop is a working intermediate, not evidence — never write it to the ticket's evidence folder.
Build the annotation HTML — scratchpad only. The crop goes in as a background <img>; each annotation is one absolutely-positioned <div> overlay. Do NOT design from scratch — copy/adjust the commented blocks in references/shapes.html (circle-callout, arrow-to-region, callout-box, badge-corner, axis-tick + axis-tag, before-after) and keep its z-index scale.
Serve it over loopback. The browser-automation CLI refuses file:// URLs (errors out before rendering) — a local HTTP server is the only way, and loopback binding is also what keeps this approach exfiltration-safe, not just a workaround:
```
python3 -m http.server <port> --bind 127.0.0.1 --directory <scratchpad>
```
Capture with the browser-automation CLI. Load /playwright-cli first (CLI → skill auto-load rule) for exact verbs/flags. Flow: open http://127.0.0.1:<port>/annotation.html in a named session of its own (-s=<KEY>-annotate; anonymous, case (a) of ../agentic-qa-core/references/browser-sessions.md), close it when the PNG is on disk, resize the viewport to the HTML's real dimensions (equal or larger — a smaller viewport clips callouts), then screenshot with an explicit destination path into the ticket's evidence/ folder, named:
```
{KEY}-BUG-{BUG-KEY}-annotated.png
```
{KEY} = ticket under test, {BUG-KEY} = the filed bug/defect/improvement issue key (they differ in the normal case). This is the ONE file of the whole process that belongs in evidence/.
Inspect and iterate. Read the PNG back and check it reads cleanly. Not optional and not one-shot — expect at least one adjustment pass: move/resize a circle or callout box, rewrap/shorten callout text, nudge the badge or arrow out of a collision.
Clean up. Kill the HTTP server process (kill <pid> or pkill -f "http.server <port>"). Never leave it running past the session.
Report the path — immediately, unprompted. The moment the final PNG lands in evidence/, state its repo-relative path in chat, in that same turn (standing contract: ../agentic-qa-core/references/session-footer-contract.md Part 1). The path must also reappear in the session-close consolidated screenshot list, leading the "Bug annotations" group.
(Optional — this repo's upgrade over the manual-attach model.) Embed into the bug issue. Jira accepts inline images via the bundled helper — offer to publish the annotated PNG directly as an evidence comment on the bug (human confirms first):
```
bun .agents/skills/acli/scripts/jira-attach-media.ts <BUG-KEY> \
  ./evidence/<KEY>-BUG-<BUG-KEY>-annotated.png \
  --caption "Annotated evidence — <one-line defect summary>" --publish
```
The annotated image is the "money shot": it leads the bug's Evidence section and ranks first (ahead of the raw capture) in the Stage 3 Evidence Handoff (../sprint-testing/references/reporting-templates.md §3.5).

Known gotchas

Badge hidden behind a callout box. Both are position: absolute; without explicit stacking, DOM order (not visual intent) decides paint order. references/shapes.html fixes this with an explicit z-index scale — base image 1 → circles/arrows/ticks 10 → callout boxes 20 → corner badge 30 (always topmost). Skipping the scale is the #1 source of annotation bugs; don't let a copy-pasted block quietly drop its z-index.
Callout text clipped at the edge. Happens when the resize viewport is smaller than the HTML canvas, or a callout box lacks max-width + white-space: normal. Size the viewport to the real image dimensions or larger — never smaller; keep callout boxes wrapped, not nowrap.
file:// doesn't work. Always serve over loopback HTTP (step 4).
Mojibake on em-dashes / accented characters. Without <meta charset="utf-8"> in <head>, — and accents render as garbage when captured. references/shapes.html already carries the meta tag — keep it when copying.

Evidence rules

Binding conventions: ../agentic-qa-core/references/evidence-conventions.md — bucket model (§1: intermediates are Bucket C, final PNG is Bucket B), annotated-name suffix (§2), verify-on-disk discipline (§3).