e2e-test-hardener

E2E Test Hardener

Turn generated/authored Playwright E2E tests into mergeable, robust tests. The key is not CLI syntax but harness thinking: what to test, what to mock, and when to verify stability.

This skill applies to code the three design decisions that the Naver Financial presentation "Playwright E2E test harness for AI agents" shared as "decisions other teams could reference."

Premise: This is not a Playwright tutorial

Browser launching, page.route, storage state save/restore, selector syntax, etc. — CLI/API mechanisms are playwright-cli already in the skill. Do not explain them redundantly; refer to that instead:

• Network mocking (page.route, conditional responses, response modification): .codex/skills/playwright-cli/references/request-mocking.md
• Auth/session state save/restore (storage state): .codex/skills/playwright-cli/references/storage-state.md
• How semantic selectors come out via codegen: .codex/skills/playwright-cli/references/test-generation.md

The value this skill adds is judgment about when and why to use those tools.

When to use

• When codegen or an AI agent has just generated E2E tests and they need to be made robust before merging
• When tests fail "sometimes" (flaky). Catching it right after writing is overwhelmingly cheaper than tracking it down after merging
• When every test re-traverses the same common front part such as terms agreement/login, making them slow and causing cascading failures
• When uncontrollable external calls such as Naver certificate/HomeTax/electronic certificate/external app integration are mixed into the tests

When not to use: Pure unit tests (Jest/Vitest), one-off browser automation scripts (data extraction/screenshots only), contract tests that genuinely verify the external system itself.

The three principles at a glance

A. 테스트 독립성   — 검증할 갈래의 '시작점'에서 출발. 공통 앞부분은 테스트용 API로 미리 세팅.
B. 외부 의존성 모킹 — '통제할 수 있는 것만 테스트'. 호출 위치에 따라 모킹 지점이 갈린다.
C. 플래키 예방      — 작성 단계에서, 머지 전에 잡는다. 하드 대기·취약 셀렉터 제거 + 번인.

The three principles have an order. Narrow down what the test is responsible for with A → fix uncontrollable inputs with B → eliminate the remaining nondeterminism with C. If you do only C, you will forever chase the flakiness of a giant test.

Workflow

When given the target test(s), pass them through the three principles in order. Each step takes the form "find → fix → rationale."

Step 1 — Test independence (A)

Goal: Each test starts from the starting pointof the branch it verifies. Every test does not re-traverse the common front part (terms agreement/mobile phone verification/login) through the UI.

Find — scan for the following signals:

• Multiple tests repeat the same login/terms/verification UI steps by copy-paste
• Tests depend on execution order (data created by an earlier test is used by a later test)
• one giant test() verifying 5 or more steps at once inside
• beforeEachclicks the login form via UI every time

Fix it — move the common prefix to externalize it as test setup. Priorities:

1. test-only setup APIif available, create the state in advance with it (issuing accounts with terms agreed and authentication completed, etc.). Then verify via UI starting from the next branch.
2. Create the authentication state once as storage state and reuse it → playwright-cli's .codex/skills/playwright-cli/references/storage-state.md(state-save/state-load) pattern.
3. Tests must be independent of each other. Cut off shared state and execution-order dependencies.

Rationale — repeating the common prefix (1) accumulates execution time and (2) if the prefix breaks, all branches fail in a chain and the real cause gets buried. Playwright's official recommendation is also "each test should be completely independent." The deep judgment (the boundary of how far to set up via API and from where to verify via UI) is references/independence.md.

Step 2 — Mocking external dependencies (B)

Goal: 'Test only what you can control.' Run our code's branches for real, but replace uncontrollable externals (Naver certificate/HomeTax/electronic certificate/Naver app integration, payment PG, external OAuth, etc.) with fixed responses.

Key judgment — where do external calls happen:

브라우저(클라이언트)에서 호출  →  Playwright page.route 로 응답 가로채기
                                  (references: playwright-cli/request-mocking.md)

서버에서 호출                   →  page.route로 못 잡는다.
(Next SSR / BFF / route handler)   E2E 전용 환경변수로 고정 응답 반환하도록 분기
                                  (references: external-mocking.md)

If you get this branch wrong, the mocking "looks like it works" but actually hits the real external (slow, and our tests break when the external changes). In a Next.js enterprise app, the same data can come via SSR/BFF or via client fetch, so check the call stack first. For the judgment criteria, environment-variable gating pattern, and the "externals that must truly be mocked" checklist, see references/external-mocking.md.

Rationale — calling externals as-is makes tests slow, and our tests turn red because of changes in external systems we didn't change. Tests should verify only the behavior we controlso the signal stays clean.

Step 3 — Flaky prevention (C)

Goal: eliminate nondeterminism at the writing stage, and confirm with a pre-merge burn-in. Look at three things.

C-1. Remove hard waits. waitForTimeout, sleep, fixed setTimeout waits are prohibited. Replace with condition-based web-first waits (only as much as needed):

// 나쁜 예 — 시간에 베팅. 느린 CI에서 터지고, 빠를 땐 시간 낭비
await page.waitForTimeout(3000);
await expect(page.getByText('완료')).toBeVisible();

// 좋은 예 — 조건이 만족되면 즉시 진행, 아니면 타임아웃까지 자동 재시도
await expect(page.getByText('완료')).toBeVisible();

C-2. Semantic selectors. Find by role+name (accessibility tree) instead of CSS class names, DOM structure, or nth-child. Class names change silently with refactoring or style changes, but roles/names are the meaning the user sees, so they are stable:

// 나쁜 예 — 구조/클래스에 결합. 마크업 바뀌면 깨짐
await page.locator('.btn-primary.submit-btn').click();
await page.locator('div > form > button:nth-child(2)').click();

// 좋은 예 — 사용자가 인지하는 역할+이름
await page.getByRole('button', { name: '다음' }).click();

C-3. Burn-in. Immediately after writing, run repeatedly with --repeat-each. If it fails even once, treat it as flaky and do not merge:

# 새로 만지거나 추가한 테스트만 골라 10회 반복
npx playwright test path/to/changed.spec.ts --repeat-each=10

# 워커 1개로 돌리면 순서 의존/공유 상태 결함이 더 잘 드러난다
npx playwright test path/to/changed.spec.ts --repeat-each=20 --workers=1

Rationale — Flaky tests tracked after merge are hard to trace to who, when, and why they broke, and they erode CI trust. The cost of catching them immediately after authoring, while the author still holds all the context, is overwhelmingly cheaper. The full list of anti-patterns, the web-first wait mapping, and the burn-in round guide are references/flaky-prevention.md.

Final checklist

After polishing the tests, they must pass the following to be called "robust." The rationale and details for each item are in references/hardening-checklist.md.

[A] 각 테스트가 검증 갈래의 시작점에서 출발한다 (공통 앞부분 UI 반복 없음)
[A] 테스트끼리 실행 순서·공유 상태에 의존하지 않는다
[B] 통제 불가능한 외부 호출이 모두 모킹/고정됐다
[B] 모킹 지점이 호출 위치(클라이언트 page.route vs 서버 env)에 맞다
[C] waitForTimeout/sleep 등 하드 대기가 0건이다
[C] 셀렉터가 role+name 등 시멘틱이다 (CSS 클래스/구조 결합 없음)
[C] 변경 테스트 번인(--repeat-each) 통과, 단 한 번도 실패 없음

Once the checklist fully passes, briefly report to the user what was fixed and how (by principle A/B/C) along with the burn-in results.

Reference files (read only when needed)

• references/independence.md — A: API setup vs UI verification boundary, storage state reuse pattern, order-dependency removal recipe
• references/external-mocking.md — B: Client vs server call discrimination, E2E-only environment variable gating, mocking target checklist
• references/flaky-prevention.md — C: Hard wait → web-first wait mapping table, fragile selector → semantic conversion, burn-in round/worker guide
• references/hardening-checklist.md — Pre-merge final checklist (with rationale) and user report template