rag-retrieval-testing

RAG Retrieval Test (Chinese Version)

When to Use

Need evidence-based analysis, design, or validation preparation around query variants, chunking, filtering, recall/precision proxies, ranking, freshness, and retrieval evidence.
Need to review an Agent, RAG, or LLM solution, results, or evidence, and form actionable improvement items.
Input is incomplete, but a constrained initial version with assumptions, information gaps, and human decision boundaries still needs to be delivered.

Default output is Markdown, organized by domain risk, evidence status, priority, and boundaries.
When the user requests table, CSV, JSON, or ticket format, preserve the same finding fields, evidence, and decision boundaries.
Before output enters an automated process, first confirm the schema, enum values, required fields, and evidence sources.

First read and follow prompts/rag-retrieval-testing.md, including input audit, domain coverage, and output order.
Extract scope, environment, version, time window, constraints, success criteria, and available evidence, with focus on checking query variants, chunking strategy, filter conditions, recall proxies, ranking, and freshness.
First distinguish confirmed facts, evidence-supported inferences, candidate recommendations, and Human decisions, then sort by risk and evidence strength.
Convert high-risk items into preconditions, steps, expected results or judgment criteria, required evidence, and verification methods.
When information is insufficient, deliver a constrained initial version, clarify which conclusions cannot hold, and do not write static materials as execution results.

When calibrating domain results, correct controls, or evidence gaps, read references/domain-decision-example.md 's desensitized inputs, item-by-item expectations, and recomputation basis.
Each execution must read prompts/rag-retrieval-testing.md; it is the complete execution specification of this Skill.
When evaluation is needed, read evals/eval.yaml and the matching cases under evals/cases/ .
Only read references/、examples/、scripts/ or output-formats.mdwhen the directory actually exists and the task requires it; do not assume assets that do not exist.

Fix annotation rules, document/chunk granularity, annotators, and versions for the query and related collections; the Recall@k denominator is the known relevant set, and Precision@k is calculated based on the actual number returned and the project's duplicate-chunk rule.
Verify top-k/rank, relevance, tenant/permission filtering, freshness, and document version result by result; unauthorized results cannot be counted as successful just because they are relevant, and outdated results are not offset by high scores.
Control comparisons of ranking or configuration changes with the same query, index, filtering, and resources; without a complete relevant set, report only proxy metrics and limitations, and do not claim true recall is complete.
Distinguish retrieval failure, where relevant material is not retrieved, from generation failure, where correct material is retrieved but the generation is wrong; missing generation output cannot localize the generation stage, and missing labels remain UNKNOWN.

Only analyze query variants, chunking, filtering, recall/precision proxies, ranking, freshness, and retrieval evidence; do not replace the business owner or Human in risk acceptance, exception approval, and safety decisions.
Do not fabricate system behavior, fields, model outputs, data, thresholds, root causes, execution records, or pass conclusions.
Static design, plans, file existence, or dry-run can only retain the original evidence status and cannot be upgraded to proof of real execution.
When evidence is insufficient, use pending confirmation, blocked, unassessed, or NOT_SCORED, and provide a minimal verification method.
When user data, production, or security is involved, use least privilege, desensitized data, mocks, dry-run, or an isolated environment.

Query variants, chunking strategies, filter conditions, recall proxies, ranking, and freshness are covered, with source, evidence status, and verification method recorded for each item.
Facts, inferences, candidate recommendations, information gaps, and Human decisions are separated.
High-risk items have P0/P1/P2/P3 or equivalent priority, responsible role, and closure conditions.
Plans, static checks, or dry-runs are not written as test execution, all passed, or security approval.
Residual risks, stop/escalation conditions, and next steps are stated.

Listing only checkpoints without explaining trigger conditions, expected focus points, responsible roles, closure conditions, and evidence.
Treating test or model tool names from adjacent domains as a complete judgment of retrieval results.
Using unexplained numbers to create false precision, or writing correlation as causation.
Directly refusing when the input is incomplete, or conversely pretending the conclusion is already certain.

Start from the paths most likely to cause user harm, business loss, or decision blockage.
Use the smallest verifiable experiment to reduce uncertainty, and record conditions, versions, sources, and evidence.
Enable another engineer to independently install, execute, and review the outputs of this Skill.