SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsqa-testing-performance

qa-testing-performance

Testing & quality

View repository

Updated 10-16 08:09

About this skill

QA Testing (Performance)

This skill owns load-test design, CI performance gates and capacity validation. Route diagnosis of an observed hot path to software-performance.

Sources: data/sources.json.

Workflow

  1. Establish realistic baselines, traffic models, and budgets.
  2. Choose the right performance test type and toolchain for the risk.
  3. Run warm-up separately, then repeat the scenarios against representative environments with monitoring active; preserve raw results rather than selecting the best run.
  4. Compare only runs whose workload, data volume, cache state, build, infrastructure allocation, generator capacity, network path, and telemetry query are known. Report a distribution or confidence interval so run noise is visible.
  5. Analyze bottlenecks, then convert findings into budgets, profiles, and CI gates. Label evidence synthetic, representative, or production-observed; a budget checker validates supplied measurements, not environment parity or capacity beyond the tested concurrency, duration, and faults.

Inputs to Gather

  • SLOs/SLAs for latency, availability, throughput
  • Critical user journeys and their expected traffic volumes
  • Infrastructure topology (services, databases, caches, queues, CDN)
  • Current performance baselines (if any)
  • Traffic patterns: peak hours, seasonal spikes, growth projections
  • Database query hotspots and slow query logs
  • Frontend targets: Core Web Vitals (LCP, INP, CLS), bundle size limits
  • Environment parity: how close is the test environment to production?

Test Types

Type Purpose When to Run
Load Validate behavior under expected traffic Pre-release, nightly
Stress Find the breaking point beyond expected capacity Pre-release, capacity planning
Soak / Endurance Detect memory leaks, connection pool exhaustion, GC degradation over hours Weekly, pre-release
Spike Validate behavior under sudden traffic bursts Pre-release, after autoscaling changes
Capacity Find the ceiling — max throughput before SLO violation Quarterly, before major launches

When NOT to Load Test

Load testing has a real cost (build time, environment risk, engineer attention). Skip or downscope it when:

  • No SLO exists yet and traffic is trivial. An internal tool with 5 users and no growth path doesn't need a load-testing harness — define the SLO first, or skip until one exists.
  • Production telemetry already answers the question. If RUM/APM already shows the system holding p95/p99 comfortably under real peak traffic, a synthetic load test adds little; spend the budget on a soak test only when telemetry leaves a leak or GC-drift hypothesis unresolved.
  • The bottleneck is already known and reproducible. Don't stand up a load-testing campaign to "confirm" a bug you can already reproduce with a single request under a profiler. Profile, fix, then run one confirmation load test — not a full suite.
  • Pre-product-market-fit. Traffic shape, critical journeys, and even the API surface are still changing weekly. A load test built now models a system that won't exist in a month; capacity arithmetic is a preliminary sanity check, not evidence of headroom or tail latency.
  • Third-party or partner APIs without an agreement to test. Load testing a vendor's production endpoint without permission risks rate-limit bans or ToS violations. Use their sandbox/staging tier, or model the dependency's expected latency into your own capacity math instead of hitting it directly.
  • A one-off batch job with no user-facing latency SLO. Throughput/capacity math (rows/sec needed vs. measured single-run rate) is usually sufficient; a full VU-ramp load test is overkill for a job that runs once a night with no concurrent users.

When in doubt, the cheapest sanity check is Little's Law arithmetic (see references/capacity-planning.md) — if the back-of-envelope math shows huge headroom, a full load-testing campaign may not be worth the cost yet.

Tool Selection

Tool Language Strengths Best For
k6 JavaScript Arrival-rate executors and threshold gates. Verify browser APIs, assertions, extensions and operator compatibility in the installed-version docs before choosing a pin. API/browser load testing, CI gates
Locust Python Distributed, flexible, real Python scripts Complex scenarios, Python teams
Artillery YAML + JS Declarative, good for APIs, scenario chaining API load testing, quick setup
Gatling Java/Kotlin/Scala/JS/TS Strong reporting, enterprise support; JS/TS support (GraalVM) JVM teams, enterprise
JMeter Java/GUI Widely adopted, protocol support Legacy, protocol-heavy testing
Playwright JavaScript Real browser, network interception Frontend performance, E2E perf
Lighthouse CI JavaScript Lab performance audits, accessibility, SEO Frontend budgets, PR gates

SLO-Driven Test Design

Design the test from the SLO, not the other way around. Working backward from "let's load test and see what we get" produces budgets that are either too loose (numbers the system already hits) or arbitrary (nobody can say why 500ms and not 600ms).

  1. Start from the availability/latency SLO. Example: 99.9% of requests must complete under 300ms, measured over a rolling 30-day window.
  2. Keep the error-budget denominator explicit. For a request-based 99.9% latency SLO, at most 0.1% of eligible requests may breach its target over the chosen window. Minutes of downtime apply to a time-based availability SLI, not automatically to a request-based latency SLO (SRE SLI definitions).
  3. Set the load test's arrival rate at expected peak, not average. If peak is 500 rps, test at 500 rps (open-loop/arrival-rate — see references/load-testing-patterns.md), not at a comfortable average that never stresses the tail.
  4. Choose CI margin from baseline variance and release risk. A tighter CI threshold can warn before the production SLO is breached; a fixed percentage is a policy example, not a universal requirement. Preserve the production SLI denominator and percentile when translating the target.
  5. Goodput, not just percentiles, for SLO-gated systems. Track the fraction of requests meeting the SLO at the tested concurrency (goodput) alongside p50/p95/p99 — a system can have a "fine" p99 while goodput is dropping because the failure mode is errors, not just slow responses. Whether percentiles include failed requests is tool-dependent — k6's http_req_duration includes failed requests by default, so fast failures can pull percentiles down and mask the real problem; verify your tool's convention before trusting a percentile alone.

CI Integration Patterns

Performance budgets — define explicit thresholds:

  • API latency: p95 < target, p99 < ceiling
  • Throughput: requests/sec >= minimum under load
  • Error rate: < threshold (example budget: 0.1-1%; set the actual number from your own SLA/error budget, not this range)
  • Core Web Vitals: look up the official good thresholds before choosing field budgets. Assess field p75 separately from lab runs; CrUX uses its reporting window, while RUM windows are configured by the team
  • Bundle size: < budget per entry point
  • LLM endpoints: TTFT low enough for the target UX (example budget: <500ms for chat; set your own from UX research, not this figure), tokens/sec >= target, goodput >= SLO at peak concurrency

When to run each tier:

  • PR gate (lightweight): Lighthouse CI, bundle size check, smoke load test (30s, low VUs)
  • Nightly (full): Full load test suite, soak test (1-2 hours), baseline comparison
  • Pre-release (capacity): Stress test, spike test, capacity test with production-like data

Artifact collection: test results JSON, flamegraphs, comparison reports, Grafana snapshots, Lighthouse HTML reports.

Frontend, Backend, Profiling, and Capacity

Core Web Vitals (LCP/INP/CLS), Lighthouse CI budgets, bundle-size tracking, and synthetic-vs-RUM monitoring — see references/frontend-performance.md.

Connection-pool pressure and read-replica lag testing under load — see references/database-performance-testing.md. Query-level diagnosis (EXPLAIN ANALYZE, index effectiveness, N+1 detection) is owned by data-sql-optimization.

How profiling connects to a load test (always profile under realistic concurrency, compare against a pre-change baseline) — see references/profiling-optimization.md. Per-language profiler commands are owned by software-performance.

Sizing instances from stress-test results, cost modeling, and auto-scaling validation — see references/capacity-planning.md.

Quick Reference

Task Approach Key Metric
API latency regression k6 with baseline comparison p95 delta
Frontend speed regression Lighthouse CI lab budget plus field RUM Lab LCP/CLS/TBT; field LCP/INP/CLS
Memory leak detection Soak test + heap snapshots Heap growth over time
Capacity ceiling Stress test with ramp-up Max RPS before SLO breach
Database bottleneck Query benchmark + connection pool test Query p95, pool wait time
Bundle size regression size-limit or bundlemon in CI Bundle size delta
Auto-scaling validation Spike test with monitoring Scale-up latency, error rate during scale
LLM/AI API latency Separate TTFT and generation throughput; test with realistic prompt distributions TTFT p95, tokens/sec, goodput
Continuous profiling Pyroscope (Grafana) for always-on production profiling Regression detection at deploy time

Decision Tree

Performance concern: [Symptom]
    │
    ├─ Slow API responses?
    │   ├─ Under normal load? → Profile backend (flamegraph + trace spans)
    │   └─ Only under load? → Load test → find bottleneck (CPU, DB, pool, GC)
    │
    ├─ Slow page load?
    │   ├─ Large bundle? → Bundle analysis + code splitting
    │   └─ Slow server response? → API profiling + caching
    │   └─ Layout shift? → CLS debugging (font, image, dynamic content)
    │
    ├─ Unknown capacity?
    │   └─ Stress test → find ceiling → capacity plan with headroom
    │
    ├─ Memory growth over time?
    │   └─ Soak test + heap snapshots at intervals → compare retained objects
    │
    ├─ Latency spikes under traffic bursts?
    │   └─ Spike test → check auto-scaling, connection pools, queue depth
    │
    ├─ LLM/AI endpoint?
    │   ├─ TTFT too high? → prefill optimization (prompt length, batching)
    │   ├─ Throughput degrading? → tokens/sec sweep + concurrency step test
    │   └─ Inconsistent under load? → warm vs cold cache scenario; mock for CI
    │
    └─ Need CI performance gate?
        ├─ API? → k6 thresholds in pipeline
        ├─ Frontend? → Lighthouse CI budgets
        └─ Bundle? → size-limit or bundlemon

Resources

Resource Purpose
references/load-testing-patterns.md Load test design: scenarios, ramp-up, data, analysis; open/closed loop; LLM/AI API testing
references/performance-budgets-ci.md CI performance gates and baseline management
references/profiling-optimization.md Profiling under load, network latency decomposition, continuous profiling with Pyroscope; per-language profiler commands owned by software-performance
references/frontend-performance.md Core Web Vitals, Lighthouse CI, bundle tracking
references/database-performance-testing.md Connection-pool pressure and replica-lag testing under load; query-level diagnosis owned by data-sql-optimization
references/capacity-planning.md Load results to infrastructure sizing

Data

File Purpose
data/sample-perf-results.json Synthetic B2B SaaS performance test results: budgets, measured values, and test scenarios

Templates

Template Purpose
assets/template-performance-test-plan.md Performance test scope, scenarios, and acceptance criteria
assets/template-k6-load-test.js Starter k6 script with stages, thresholds, and custom metrics
assets/template-performance-budget.md Budget definition for latency, throughput, and Core Web Vitals

Scripts

Stdlib-only Python CLI — no external dependencies, runs with Python 3.10+.

Script Purpose
scripts/perf_budget_checker.py Budget validation, CI tier planning, and full Markdown report generation
scripts/test_perf_budget_checker.py Offline regressions (pytest and Node.js required); run python3 -m pytest scripts/test_perf_budget_checker.py -q

Run from the qa-testing-performance/ directory:

# Budget check — PASS/WARN/FAIL per metric, overall CI gate verdict
python scripts/perf_budget_checker.py check --input data/sample-perf-results.json

# CI test tier assignment (PR_gate / nightly / pre_release) for all scenarios
python scripts/perf_budget_checker.py plan --input data/sample-perf-results.json

# Full Markdown performance test report to stdout
python scripts/perf_budget_checker.py report --input data/sample-perf-results.json

# Write report to file
python scripts/perf_budget_checker.py report \
  --input data/sample-perf-results.json \
  --output report.md

The check and report subcommands exit 1 on FAIL, 0 on PASS or advisory WARN, and 2 on invalid input. A nonempty budget and every budgeted measurement are required; plan also requires at least one valid scenario. WARN zones are local heuristics that permit small breaches; use explicit k6/LHCI error thresholds for a strict gate. See scripts/README.md for full format details and threshold reference.

Navigation

  • ## Workflow and ## Decision Tree for the baseline sequence and test-type selection
  • ## When NOT to Load Test before committing to a full test campaign
  • ## Resources, ## Templates, and ## Scripts for deeper materials and automation
  • ## Related Skills for observability, resilience, and diagnosis handoffs

Related Skills

Skill Purpose
qa-observability Metrics, tracing, and performance monitoring
qa-resilience Failure mode testing under load
qa-testing-strategy Risk-based test strategy
qa-debugging Performance debugging and profiling
software-performance Diagnosing an observed regression: profiling, misdiagnoses, tail-latency math, neutral-is-a-revert
ops-devops-platform CI/CD and infrastructure
software-backend Backend API optimization
data-sql-optimization Database query tuning

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.