Accessibility testing automation, CI gating, and audit methodology for web and mobile: what to automate, what needs manual testing, and how to keep a sustainable gate. This skill covers testing and conformance evidence; software-ui-ux-design covers accessible design and software-accessibility covers implementing fixes.
Workflow
- Gather the target surfaces, WCAG scope, and platform constraints.
- Cover what automation can reliably detect first.
- Add manual keyboard and screen-reader checks for critical journeys.
- Turn findings into CI gates, remediation work, and verification evidence.
Inputs to Gather
- Target WCAG level: AA (default) or AAA (specific requirements)
- Platforms: web, iOS, Android, or combination
- Assistive technology requirements: screen readers, switch access, voice control
- Existing accessibility tooling and baseline violation count
- Regulatory requirements: ADA Title II, Section 508, EN 301 549, EAA — see
## Regulatory Landscape; do not assume any single deadline applies to a private US business - Critical user flows that must be accessible
- Design system or component library status (tested vs untested)
WCAG 2.2 Automation Boundary
Automation covers only a subset of WCAG 2.2. The exact share varies by product, framework, and tool rule set, so do not treat any percentage as a conformance rule.
| Coverage Type | Examples |
|---|---|
| Reliably automatable | Missing alt attributes, missing form labels, many ARIA validity checks; candidate contrast, heading and landmark findings still need scope review |
| Candidate detection + human verification | Alt-text quality, keyboard usability, touch target adequacy, focus order, live-region behavior |
| Manual only | Cognitive load, reading level, error-prevention UX, consistent navigation, timing adjustments, motion preferences, full screen-reader usability, reflow at high zoom |
Read references/wcag-automation-matrix.md when you need criterion-level automation planning beyond the table above.
Canonical axe tag set
Use this array everywhere axe-core runs (Playwright, Cypress, jest-axe, Storybook, the baseline script):
['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa'] // WCAG 2.2 Level A + AA
// optional: add 'best-practice', and report those results as non-WCAG
Why all five: axe gives each rule exactly one version/level tag, the WCAG version that introduced its criterion (axe-core API: tags). wcag2aa means "WCAG 2.0 AA rules", not "all AA rules". ['wcag2a','wcag2aa','wcag22aa'] silently skips the 2.1 rules — for example autocomplete-valid (1.3.5) and avoid-inline-spacing (1.4.12) — so a gate stays green against a WCAG 2.1 AA legal standard it never tested. Check the installed axe rule metadata for default-enabled and experimental rules; runOnly tag selection changes which rules run. No tag set equals conformance.
Tools for a second rule engine: Lighthouse (axe-based, weighted score), Pa11y (runners: ["axe","htmlcs"]), IBM Equal Access; WAVE for manual review. See references/automated-auditing.md.
Conformance Boundary
Automated scans and tool severity labels do not establish WCAG conformance; they are implementation guidance. A conformance claim needs combined evidence: automated findings, manual review, and assistive-technology testing on the relevant journeys.
Accessibility Overlays: Do Not Recommend
Overlay/widget products (one injected script that claims to auto-remediate) do not confer conformance and add legal exposure: the FTC ordered accessiBe to pay for unsubstantiated compliance claims. Read the cited report before quoting litigation figures; a share of filings does not establish a higher lawsuit rate among overlay users. Record an installed overlay as an open risk, not a control, and plan remediation plus removal. Evidence: references/regulatory-landscape.md.
Prioritizing Fixes: User Impact, Not Violation Count
Violation count is a weak signal for triage. Rank by: (1) absolute barrier vs. workaround — a missing label on the only checkout submit button outranks fifty redundant footer roles; (2) flow centrality — sign-in, checkout, account recovery, and anything users cannot route around; (3) breadth — a shared component or layout template defect hits every consuming page. Two critical violations on a conversion path outrank 200 minor ones; do not let count-sorted dashboards drive the backlog.
Design-System Leverage
Fix shared components (Button, Modal, Form Field, Tabs, Combobox) first: one fix remediates every consuming page. Audit core interactive components against their ARIA APG pattern, fix, then re-scan pages — many page-level findings collapse. Component-level tests (Storybook a11y with parameters.a11y.test = 'error', jest-axe) stop regressions in these primitives.
Web Testing Patterns
- Component tests: axe-core via
@axe-core/playwright,cypress-axe, or Storybooka11yaddon. - E2E tests: inject
@axe-core/playwrightinto page-level assertions after navigation. - Lighthouse CI: the category score is weighted and can pass a serious failure; block on named audits, track the score as a trend.
- States: axe skips hidden regions. Open menus, dialogs, error states, and themes, then re-scan; re-scan after SPA route changes.
incompleteresults are needs-review items: route to manual review, never block on them or count them as passes.
Read references/automated-auditing.md when wiring axe-core, Lighthouse, or Pa11y into component or CI tests.
Mobile Testing Patterns
- iOS:
XCUIApplication.performAccessibilityAudit(for:_:)in UI tests (Xcode 15+, iOS 17+), Accessibility Inspector for exploratory audits, VoiceOver manual protocol.accessibilityIdentifieris a test hook VoiceOver never reads, so identifier lookups are not accessibility checks. - Android: Compose
composeTestRule.enableAccessibilityChecks()(Compose 1.8.0+,ui-test-junit4-accessibility), EspressoAccessibilityChecks.enable(), Accessibility Scanner, TalkBack manual protocol. - Map WCAG to native apps with W3C WCAG2ICT.
Read references/mobile-accessibility.md when scoping iOS or Android accessibility test workflows.
CI Integration
| Stage | What to Run | Gate |
|---|---|---|
| PR (component) | axe-core on changed components | Use team policy; common starter is block critical/serious |
| PR (E2E) | axe-core on smoke flows | Use team policy; common starter is block critical/serious |
| Staging deploy | Full-page Lighthouse + axe scan | Common starter: block critical, warn serious/moderate |
| Release | Manual audit of critical flows + screen reader verification | Sign-off required |
Baseline management: for existing codebases, snapshot current violations and gate only on new ones. Compare per-node fingerprints (rule + page + target), not per-rule node counts: with a count diff, one fixed node hides one new node of the same rule. See references/ci-accessibility-gates.md.
Close a finding against the criterion, assistive-technology/browser setup, viewport/device, and user task that exposed it. Preserve the failing evidence, retest the repaired component or flow, then check adjacent focus, announcements, keyboard order, zoom/reflow, and error recovery as applicable. A clean automated rerun proves only the rules executed on the scanned state; record which manual, expert, or real-user tier remains untested.
Three Tiers of Testing: Automated, Manual Expert, Real AT Users
Each tier catches defects the others miss — none is a substitute for the others on a compliance-relevant or high-traffic critical flow:
| Tier | Who | Catches | Misses |
|---|---|---|---|
| Automated | CI (axe-core, Lighthouse, jsx-a11y) | Structural/programmatic defects reliably, with repeatable coverage of the scanned states | Anything requiring judgment about meaning, sufficiency, or usability |
| Manual expert | Sighted engineer/QA running keyboard + screen-reader protocol | Flow-level defects, focus management, ARIA correctness in context | Real-world friction — expert testers know the workarounds a first-time AT user does not |
| Real AT users | Actual disabled users, their own devices/AT/versions/speed | Highest-fidelity signal — muscle-memory shortcuts, unfamiliar error recovery, device-specific quirks | Coverage depends on recruited users and tested tasks; combine with structural checks |
Automated plus in-house expert is necessary but not sufficient for a critical or regulated flow. Before a compliance-relevant release (EAA, ADA Title II, VPAT/ACR sign-off) or a core-flow redesign, run at least one round with actual AT users (research panels, disability ERGs, screen-reader-using customers), paid as research participants. Vendor names and panel pricing change — verify before budgeting.
Screen Reader Testing Protocol
Test critical flows with at least one screen reader per target platform:
| Platform | Screen Reader | When |
|---|---|---|
| macOS/iOS | VoiceOver | Default for Apple targets |
| Windows | NVDA (free); add JAWS for enterprise/US-heavy audiences | Default for Windows web |
| Android | TalkBack | Default for Android targets |
What to verify: landmark announcements, heading navigation, form label association, error announcements, dynamic content updates (live regions), modal/dialog focus management, custom widget interaction.
Test NVDA and JAWS both for enterprise or US-heavy audiences: JAWS still leads primary desktop use in North America (WebAIM Screen Reader Survey #10; check webaim.org for a newer survey before quoting shares).
Read references/screen-reader-testing.md when running VoiceOver, NVDA, JAWS, or TalkBack verification.
Keyboard Navigation Testing
Tab order, visible focus, skip link, modal focus trap and return, and APG keyboard patterns for custom widgets. See references/keyboard-navigation.md.
Quick Reference
| What to Test | Tool / Method | CI Stage | Automation |
|---|---|---|---|
| Color contrast | axe-core + manual for unsupported states/backgrounds | PR + release | Partial |
| Missing alt text | axe-core | PR gate | Full |
| Form labels | axe-core | PR gate | Full |
| Heading hierarchy | axe-core + manual structure review | PR + release | Partial |
| ARIA validity | axe-core | PR gate | Full |
| Keyboard navigation | axe-core + manual | PR + release | Partial |
| Touch target size | axe-core + manual | PR + release | Partial |
| Screen reader flow | Manual (VoiceOver/NVDA/TalkBack) | Release | Manual |
| Cognitive load | Manual review | Release | Manual |
| Reflow at 400% zoom | Manual browser test | Release | Manual |
Regulatory Landscape
Summary; details, sources and caveats in references/regulatory-landscape.md:
- WCAG 2.2 AA is the default target. It added six A/AA criteria over 2.1 (2.4.11, 2.5.7, 2.5.8, 3.2.6, 3.3.7, 3.3.8); 4.1.1 was removed in the 5 Oct 2023 Recommendation. Check the W3C publication status before discussing WCAG 3.0; use a published Recommendation for conformance gates.
- ADA Title II: WCAG 2.1 AA; the 2026 Interim Final Rule moved the dates to 26 Apr 2027 (50k+) and 26 Apr 2028. Title III has no DOJ technical standard or deadline.
- Section 508: still WCAG 2.0 AA.
- EN 301 549: read the applicable Official Journal citation and its directive scope before asserting presumption of conformity; publication alone does not make a newer edition the harmonised reference.
- EAA: applies from 28 Jun 2025; whether it covers a given product is a question for qualified counsel. The Art. 32 transition covers products already in use and pre-existing contracts, not "existing services" in general; microenterprises providing services are exempt; an Annex V accessibility statement is required.
Do / Avoid
Do
- Start with axe-core as the default cross-framework scanner
- Treat CI severity thresholds as team policy, not as a WCAG verdict
- Test with real assistive technology for critical flows, including sessions with actual disabled AT users before a compliance-relevant or core-flow release
- Fix shared design-system components first — highest leverage per hour spent
- Include accessibility in definition of done
- Use ARIA Authoring Practices Guide (APG) for custom component keyboard patterns
- Combine multiple automated tools for broader rule coverage
- Prioritize by user impact (blocker vs. workaround, flow centrality) over raw violation count
Avoid
- Treating automated scanning as complete coverage or as proof of WCAG conformance
- Recommending or shipping an accessibility overlay/widget as a substitute for code-level remediation — it does not confer conformance
- Blocking all PRs on all existing violations (use baselines)
- Using only one automated tool without manual testing
- Testing accessibility only before release (shift left)
- Sorting remediation backlogs by violation count alone instead of user impact
- Blocking CI on axe
incompleteresults, or disablingcolor-contrastglobally to silence noise - Gating on the Lighthouse accessibility score alone
- Writing VPAT/ACR "Supports" from automated results without manual and AT evidence
Navigation
## Workflowand## WCAG 2.2 Automation Boundaryfor the baseline sequence## Resourcesand## Templatesfor deeper materials## Related Skillsfor cross-platform handoffs
Resources
| Resource | Purpose |
|---|---|
| references/automated-auditing.md | axe-core, Lighthouse, Pa11y integration and configuration |
| references/screen-reader-testing.md | VoiceOver, NVDA, TalkBack testing protocols |
| references/wcag-automation-matrix.md | WCAG 2.2 AA criteria mapped to automation coverage |
| references/ci-accessibility-gates.md | CI gate design, baselines, and severity mapping |
| references/cognitive-accessibility.md | WCAG 2.2 cognitive criteria (3.2.6, 3.3.7, 3.3.8), reading-level checks, COGA guidance |
| references/eslint-jsx-a11y-integration.md | eslint-plugin-jsx-a11y wiring, custom rule set, CI gate with --max-warnings=0 |
| references/mobile-accessibility.md | iOS performAccessibilityAudit, Compose/Espresso checks, VoiceOver/TalkBack protocols |
| references/regulatory-landscape.md | WCAG versions, ADA, Section 508, EN 301 549, EAA, overlay evidence |
| references/keyboard-navigation.md | Keyboard testing and ARIA APG patterns |
| data/sources.json | Curated external sources |
Scripts
| Script | Purpose |
|---|---|
| scripts/generate-a11y-baseline.ts | Playwright + axe-core baseline generator; rejects malformed configuration and unsuccessful navigation before replacing the baseline — run with npx tsx scripts/generate-a11y-baseline.ts |
| scripts/test_baseline_regressions.py | Offline mocked regressions; run with Node 24+ available: python3 scripts/test_baseline_regressions.py |
| scripts/README.md | Setup and usage guide for all scripts |
Templates
| Template | Purpose |
|---|---|
| assets/template-accessibility-audit.md | Accessibility audit scope, findings, and remediation plan |
| assets/template-accessibility-ci-config.md | Example CI configurations for axe-core and Lighthouse gates |
| assets/template-screen-reader-checklist.md | Per-flow screen reader testing checklist |
Related Skills
| Skill | Purpose |
|---|---|
| software-accessibility | Implementing accessibility fixes in code |
| software-ui-ux-design | Accessible design and WCAG 2.2 design patterns |
| qa-testing-strategy | Risk-based test strategy and coverage planning |
| qa-testing-playwright | E2E web testing with accessibility assertions |
| qa-testing-android | Android accessibility checks with Espresso |
| qa-testing-ios | iOS accessibility testing with XCTest |
| qa-testing-mobile | Cross-platform mobile accessibility |
| software-frontend | Frontend development and semantic HTML |
Learnings Loop
When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.
After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.