rag-quality-testing

RAG Quality Testing (Chinese Version)

When to Use

When evidence-based analysis, design, or verification preparation is needed around RAG output grounding, relevance, completeness, citation/support, abstention, and answer-level evidence.
When an Agent, RAG, or LLM solution, result, or evidence needs review, and actionable improvement items need to be formed.
When input is incomplete but a constrained initial version with assumptions, information gaps, and human decision boundaries still needs to be delivered.

Default output is Markdown, organized by domain risk, evidence status, priority, and boundaries.
When the user requests table, CSV, JSON, or ticket format, retain the same finding fields, evidence, and decision boundaries.
Before output enters an automated process, first confirm schema, enum values, required fields, and evidence sources.

First read and follow prompts/rag-quality-testing.md, including input audit, domain coverage, and output order.
Extract scope, environment, version, time window, constraints, success criteria, and available evidence, focusing on grounding, relevance, completeness, citation support, and abstention.
First distinguish confirmed facts, evidence-supported inferences, candidate recommendations, and Human decisions, then sort by risk and evidence strength.
Convert high-risk items into preconditions, steps, expected results or judgment criteria, required evidence, and verification methods.
When information is insufficient, deliver a constrained initial version, clarify which conclusions cannot hold, and do not write static materials as execution results.

When calibrating domain results, correct comparisons, or evidence gaps, read references/domain-decision-example.md 's desensitized input, item-by-item expectations, and recalculation basis.
Each execution must read prompts/rag-quality-testing.md; it is the complete execution specification of this Skill.
When evaluation is needed, read evals/eval.yaml and the matching cases under evals/cases/ .
Only read references/、examples/、scripts/ or output-formats.mdwhen the directory actually exists and the task requires it; do not assume assets that do not exist.

Fix query/sample ID, knowledge base/index/retrieval/model/Prompt version; establish a query→retrieved chunk ID/version→atomic claim→citation location→support/contradiction/missing mapping, and do not judge grounding by the presence of citations.
Evaluate citation accuracy, claim support, and answer completeness separately: omitting required conditions may make an answer incomplete even if all existing claims are supported; judge by the user task and project rubric.
Use a comparison between answerable within knowledge coverage and should-abstain outside coverage, and separately judge normal task correctness, erroneous abstention, and should-abstain hallucination; abstention rate cannot replace quality.
When chunks, ground truth, access permissions, or thresholds are missing, keep UNKNOWN/NOT_RUN; distinguish retrieval defects from generation distortion, and report evidence scope and responsible stage.

Only analyze RAG output grounding, relevance, completeness, citation/support, abstention, and answer-level evidence; do not replace the business owner's or Human's risk acceptance, exception approval, and safety decisions.
Do not fabricate system behavior, fields, model outputs, data, thresholds, root causes, execution records, or pass conclusions.
Static design, plans, file existence, or dry-run can only retain the original evidence status and cannot be upgraded to proof of real execution.
When evidence is insufficient, use pending confirmation, blocked, unassessed, or NOT_SCORED, and provide the minimum verification method.
When user data, production, or security is involved, use least privilege, desensitized data, mocks, dry-run, or an isolated environment.

Grounding, relevance, completeness, citation support, and refusal to answer have been covered, and the source, evidence status, and verification method are recorded for each item.
Facts, inferences, candidate recommendations, information gaps, and Human decisions have been separated.
High-risk items have P0/P1/P2/P3 or equivalent priority, responsible roles, and closure conditions.
Plans, static checks, or dry-runs have not been written as test execution, all passing, or security approval.
Residual risks, stop/escalation conditions, and next steps have been stated.

Listing only checkpoints without explaining trigger conditions, expected focus points, responsible roles, closure conditions, and evidence.
Treating test or model tool names from adjacent domains as a complete judgment of RAG quality.
Using unexplained numbers to create false precision, or writing relevance as causation.
Directly refusing when the input is incomplete, or conversely pretending the conclusion is already certain.

Start from the path most likely to cause user harm, business loss, or decision blockage.
Use the smallest verifiable experiment to reduce uncertainty, and record conditions, versions, sources, and evidence.
Enable another engineer to independently install, execute, and review the output of this Skill.