qa-probe

Purpose

qa-probe is a locally running QA agent, provided as a Node.js command-line tool. The problem it solves is this: the dashboard loads and the program does not crash, but the table is blank and the charts show zero, making it hard for developers to quickly identify the cause. It builds a "source-aware" mapping between frontend API calls and real-time backend responses, then explains the failure reasons one by one and gives fix suggestions. Its role is to fill the layer beyond Playwright (user journeys) and Schemathesis (contract fuzzing)—prioritizing answering "why is this page empty."

Main capabilities

• Source-aware mapping: uses Babel to parse the frontend, detects axios/fetch calls and common hook patterns, and extracts routes from React Router (JSX and createBrowserRouter) and TanStack Router.
• Live probing with real authentication: supports bearer, cookie, api-key, or none; concurrency is limited, and there is a hard per-request timeout to prevent streaming endpoints from hanging the entire run.
• Root cause classification: about 25 causes, each with fix suggestions. Common causes include trailing slash mismatch (frontend /api/users/ vs backend /api/users), feature flags (such as HAS_BILLING=false) silently disabling an entire route, database connected but table has no data, backend refactoring renamed fields while the frontend still reads the old names, endpoints missing authentication guards or leaking data without a token, etc.
• ID chained probing: first probe collection endpoints, retrieve real ids, then use them to probe detail routes instead of using guessed values, enabled by default.
• Adaptive baselines: learn the normal latency and row counts for each endpoint from recent runs, and flag behavior that deviates from its own baseline; model-independent and fully inspectable.
• Feedback mechanism: qa-probe label or MCP tools can record judgments and reapply them in subsequent runs; scoped labels are automatically revoked when behavior changes, avoiding masking regressions.
• Security checks (optional): anonymous authentication bypass retesting, PII scanning, role-based privilege escalation matrix; GET only, no writes initiated.
• Response assertions: declare invariants (in, type, gte, pattern, present, minItems, etc.), and qa-probe verifies them on every 2xx response.
• Write flows (optional, modifies data): completeCRUD chain with guaranteed cleanup; disabled by default, only suitable for disposable/test tenant environments.
• Other: SSE and WebSocket checks, schema drift detection, blast radius, regression diff, coverage, HAR import for dynamic frontends, OpenAPI auto-discovery (tries common spec paths and supports local spec files).

How to use

After installing via npm, create a configuration file qa-probe.config.js in the project root directory (see the official repository README for documentation examples). The tool has three phases, each cached, and can be rerun independently:

1. Analyze: parse the frontend AST, extract routes, fetch the OpenAPI spec (or import HAR), and build a dependency graph with blast radius.
2. Probe: authenticate once, then perform live probing of each discovered endpoint (HTTP, SSE, WebSocket), with hard timeouts and ID chained probing.
3. Report: classify root causes, apply baselines and feedback, score routes from 0-100, write JSON / Markdown / ai-context / HTML reports (written to the .qaprobe/ directory), and output a summary table in the terminal.

The command line provides the --fail-under <N> option, which exits with a non-zero status when the total score is below N, and can be used for CI gating (for example, qa-probe run --fail-under 80).

Working with AI assistants: qa-probe comes with a Model Context Protocol (MCP) server. First run qa-probe once to populate the cache, then configure the MCP server in .mcp.json in the repository root directory, restart the assistant, and you can ask questions in natural language, such as "Why does /reports have no data?" "If the users endpoint goes down, which pages will be affected?" "List all broken routes." and mark expected results. The output of MCP tools is sanitized; SQL errors, stack traces, and table names are redacted before reaching the assistant; the raw data remains on disk. You can also directly open the directory containing .qaprobe/ in an AI coding tool and have it read ai-context.md.

Applicable scenarios

• Daily troubleshooting of "why is this page empty": one run yields the root cause and fix suggestions, with no need to dig through the network panel and logs.
• CI gating: block deployment when the score is below the threshold.
• Demo/pre-release sign-off: confirm the environment is truly usable before a customer meeting, not just returning 200.
• Detect contract drift: when a backend refactor renames fields or moves routes, point out the specific affected endpoints and frontend calls.
• Security smoke check: detect endpoints missing auth guards or leaking PII (optional, read-only).
• Logic check: assert response invariants without writing a test suite.
• AI-assisted debugging: let Claude, Cursor, or Codex query QA status and explain failures in natural language.

Limitations and Security

Known limitations: can only detect visible HTTP calls and common hook patterns; generated SDK clients, GraphQL clients, tRPC routers, and custom service layers may require adapters, or use HAR import to discover from real traffic; dynamic URLs constructed by arbitrary factory functions cannot be detected; best results require an OpenAPI spec, and headless mode without a spec loses spec-dependent rules but can still detect observed shape drift.

Security: credentials are kept in memory only during the run and are never written to any report; configuration files are executed as JavaScript, and qa-probe should only be run with trusted configuration; ignoreHTTPSErrors: true disables TLS verification and is limited to development/pre-release environments; security checks are retested with GET only and do not write; write flows are off by default and clean up after themselves, and are only for disposable environments.

License and Branding

Licensed under the Apache License 2.0 (see the repository LICENSE and NOTICE). qa-probe may be used, modified, and redistributed under the terms of Apache-2.0; copyright and trademarks remain with LS-SIEM LLP. "qa-probe", "LightShield", and "LS-SIEM" are trademarks of LS-SIEM LLP, and the Apache license does not grant permission to use these names or logos. Contributions must follow CONTRIBUTING.md and sign the Developer Certificate of Origin (git commit -s).