qai-consultant

QAI Consultant is an open-source AI agent positioned as a senior QA architect, designed to solve the problem that creating a test strategy from scratch is time-consuming and requires deep QA expertise. It combines proven QA methodologies, industry standards (ISTQB, OWASP, ISO 26262, A-SPICE), and expert knowledge to automatically generate QA documents through RAG (Retrieval-Augmented Generation).

Key capabilities:

• From an 11-question project discovery conversation, automatically generate four documents: test strategy, risk register, effort estimation report, and test plan.
• Provide an Executive Readout above the four output tabs, deterministically summarizing overall risk, QA effort range, team capacity, confidence level, and the top three risks, without additional LLM calls.
• Deterministic QA document quality review: review existing test plans/strategies/test case lists based on ISTQB/IEEE-829 scoring criteria, with optional AI narrative.
• Test results analysis: extract metrics such as flaky, consistently failing, slowest, and failure clustering from JUnit XML/CSV, and ground the risk register in real execution data.
• QA maturity assessment: deterministic TMMi process maturity signals (10 process areas, indicative levels 1-3), as well as EU AI Act Articles 9-15 readiness scoring for AI/ML projects.
• All outputs are saved as Markdown files and can be downloaded as PDF.

Knowledge base:

• ISTQB: 14 certification syllabi (CTFL, CTAL-TA, CTAL-TM, CTAL-TAE, CT-AI, etc.)
• OWASP: WSTG v4.2, MASTG, Top 10 2021
• ISO 26262: automotive functional safety (ASIL levels, HARA, V&V)
• A-SPICE: Automotive SPICE process reference model (SWE.4, SWE.5, SWE.6)
• IEEE 829: test documentation standard
• ISO/IEC 25010: software product quality model
• EU AI Act: risk tiers, provider/deployer obligations, Article 50 transparency, Articles 9-15 testing implications
• Testing methodologies: Agile, BDD/TDD, exploratory, risk-based testing, test pyramid
• Audit and process assessment: TMMi, CMMI, ISO/IEC 33002, ISO 19011,OWASP ASVS, ISO 27001, SOC 2, and public failure case studies (Knight Capital, Boeing 737 MAX MCAS, CrowdStrike 2024)
• AI test planning: 17 real AI SDLC case studies (2024-2025)
• Expert knowledge: real QA scenarios and lessons learned from practitioners

How to use:

• Online app: quality-ai-consultant.streamlit.app (no setup required)
• Local run: refer to INSTALL.md; requires four API keys (all have free tiers); LLM calls use the Mistral API (ministral-14b-2512) as the primary provider, with OpenRouter free models (Nemotron 3 Super → GLM 5.2) as automatic fallback.
• MCP server: uvx qai-consultant-mcp, no API keys or Pinecone required, runs locally, communicates via stdio (requires uv). Can be configured in Claude Desktop's claude_desktop_config.json. Provides prompts: qa_project_interview, risk_register_structure, test_strategy_structure, test_plan_structure, and tools: retrieve_qa_knowledge, review_qa_document, analyze_test_results, assess_qa_maturity.
• CLI: supports parameters such as --review, --results, --maturity.

Use cases:

• QA engineers who need structured test strategy guidance
• Engineering managers who need effort estimation and resource planning
• Development teams without a dedicated QA architect
• QA consultants who want to accelerate delivery

Limitations and caveats:

• The cloud-hosted version relies on the Mistral API and Pinecone, and requires API keys.
• The MCP server runs locally, but the ISTQB and OWASP PDFs in the knowledge base are not bundled due to licensing reasons.
• AnonymousTelemetry is enabled by default (since v3.7.0) and can be disabled via QAI_TELEMETRY=0 or DO_NOT_TRACK=1; it only sends the tool name/success/duration/category, version, and OS family, uses a randomly generated ID per server run that is not stored, and does not send query text or project details.
• Submitted text may be logged and used for training by LLM providers.
• The public app has server-side daily generation limits (a global daily cap and a per-visitor cap, as well as a limit of 3 runs per session).
• The project uses the Apache 2.0 license.
• The version history shows continuous iteration, with the latest version being v3.6.2, the main model switched to Ministral 14B, and the output budget increased to 6,500 tokens.