How to assess findings — whether from an AI reviewer, an auditor, a challenger, or any other source of AI-generated feedback. This protocol applies to every context where an agent must decide which findings are correct, which are noise, and what to do about each.
Stance
Findings are hypotheses about the work. Some will be correct and important. Some will be wrong, inapplicable, or based on incomplete understanding. Your job is to determine which is which, with evidence.
Do not default to acceptance. The path of least resistance (apply every suggestion) produces worse code than thoughtful evaluation. Equally, do not default to rejection — that wastes valid insights.
Assess each finding by merit. The question is not "is this a big issue?" or "is this convenient to fix?" The question is: is the finding correct? A correct improvement is worth implementing regardless of size. An incorrect suggestion should be declined regardless of who raised it.
Phase 1: Investigate
For each finding, build enough context to make a high-confidence judgment. Do not assess from the finding description alone.
Your investigation stance is adversarial: attempt to disprove each finding before accepting it. If the finding survives your best effort to disprove it with evidence, it is worth acting on. If you can disprove it, discard it. This counteracts the dominant failure mode — sycophantic acceptance of AI-generated feedback that degrades correct work.
Mandatory for every finding:
- Web search to verify best-practice claims, library behavior, security patterns, and framework conventions. Training data may be outdated — web search grounds you in current best practices. This is not optional.
- Codebase inspection to trace the relevant code path and verify the finding's premise against the actual implementation.
As needed:
- Spec check for design intent when the finding touches architecture or requirements
- Adjacent code for existing patterns and conventions (does the codebase already do this elsewhere?)
Investigation tracks by claim type
Route your investigation based on what the finding claims:
| Claim type | Investigation approach |
|---|---|
| Code behavior ("this function doesn't handle X") | Trace the relevant code paths — grep, read, explore. Verify against actual implementation, not description. |
| Library/API/framework ("this API was deprecated," "method X doesn't exist") | Check ~/.claude/oss-repos/ and node_modules/ for local dependency source first — actual types, interfaces, and implementations are higher-trust than docs and version-matched to the project. Then web search with version pinning (confirm version from repo's package.json/lockfile). |
| Ecosystem/best practice ("the recommended approach is X") | Check the project's own tooling first — linter configs, type-checker settings, formatter rules. The project already declared its practices by configuring its tools. Then check framework/language official docs (version-pinned). Then web search with recency for community consensus. |
| Internal coherence ("Section 3 contradicts Section 7") | Re-read the specific sections referenced. Trace downstream implications of the contradiction. |
Evidence source hierarchy
When sources conflict, trust in this order:
| Priority | Source | Trust level |
|---|---|---|
| 1st | Source code (actual implementation, types, interfaces) | Highest for understanding current behavior. Not authority for whether that behavior is correct — use official docs and tooling for correctness judgments. |
| 2nd | Official documentation (maintainer docs, API reference) | High — may be aspirational or outdated |
| 3rd | High-trust third parties (company eng blogs, conference talks, reputable publications) | High — cross-reference rarely |
| 4th | Community sources (Stack Overflow, Reddit, dev.to, Medium) | Medium — verify claims |
| 5th | General sources (tutorials, AI-generated content, undated posts) | Low — always cross-reference |
Query formulation
When searching, formulate specific queries: {tool} {version} {specific claim} {year}. Avoid vague queries ("is X bad") or opinion-seeking queries ("should I use X"). Pin versions from the actual repo, not from training data assumptions.
Depth of understanding
For load-bearing findings (findings that would change what gets built or how), the goal is not merely "is this claim true/false?" but "what is the full picture?" Understand the implications for the current approach, the pros and cons of the finding's implied recommendation, and whether the finding changes what should happen. A finding can be factually true but its implied recommendation wrong — surface that distinction.
Investigation depth
Investigate until you reach HIGH confidence — no more, no less. Some findings reach HIGH confidence immediately (you recognize the pattern, the tooling already enforces it, the docs are clear). Others require deep investigation. The reviewer's severity label does not determine your investigation depth — your confidence does.
The mandatory minimums (codebase inspection + web search) apply to every finding regardless. Beyond those, scale depth to what your confidence requires:
- If you can classify a finding with HIGH confidence after the mandatory checks, do so — don't investigate further just because the finding is labeled Critical.
- If you cannot reach HIGH confidence after the mandatory checks, investigate deeper — even if the finding is labeled Consider or While-You're-Here.
For findings that are purely editorial (wording, formatting) where correctness is not in question, the mandatory checks may be sufficient. But "the reviewer called it Minor" is never a reason to reduce investigation — a finding labeled Minor may have correctness implications that only surface under investigation.
When web search is unavailable
If no web search tool is available, cap all confidence at MEDIUM for any claim that depends on external knowledge (library behavior, framework conventions, deprecation status, best practices). Flag the limitation to the calling workflow.
Litmus test: Could you explain your reasoning to a senior engineer and defend it with evidence? "I think" or "it seems" has not met the standard. "I verified via web search that X" or "I traced the code path and confirmed Y" has.
Phase 2: Calibrate Confidence
Name your confidence before deciding. Do not proceed to a decision at LOW confidence.
| Level | Meaning | Action |
|---|---|---|
| HIGH | Specific evidence — code lines, docs, test results, web search results | Proceed to evaluate |
| MEDIUM | Plausible but depends on unverified assumptions, or a reasonable alternative interpretation exists | Investigate further if stakes warrant. If proceeding, flag the uncertainty. |
| LOW | Cannot cite specific evidence | Do not decide. Investigate more. |
Confidence ceilings: Claims about library behavior, API semantics, deprecation status, or best practices — whether from a finding, your own knowledge, or any other source — cannot reach HIGH without external verification (web search, official docs, or codebase inspection). Training-data knowledge is not verification.
| Claim category | Confidence ceiling without external verification |
|---|---|
| Library API claims | MEDIUM max |
| Framework directives | MEDIUM max |
| Deprecation claims | MEDIUM max |
| Version-specific behavior | LOW (unless version confirmed from repo) |
| Security advisories | LOW (may be patched) |
| Best practice assertions | MEDIUM max |
When uncertain between two confidence levels, prefer the lower level. Overconfident classifications (HIGH on wrong assessments) cause more downstream damage than underconfident ones (MEDIUM on correct assessments).
Phase 3: Evaluate
For each finding, assess against relevant dimensions:
| Dimension | Question |
|---|---|
| Validity | Is the premise correct? Does the issue the finding identifies actually exist in this work? |
| Correctness | Is the proposed fix actually correct, or does it introduce new problems? |
| Applicability | Does this apply in this specific context, or is it generic advice that doesn't fit here? |
| Relevancy | Is this relevant to the scope and intent of the current changes? |
| Tradeoffs | Does applying this improve one thing but degrade another? |
| Side effects | What else changes if you apply this? Other code paths, tests, consumers? |
Not every dimension applies to every finding. A simple naming suggestion doesn't need a tradeoff analysis. An architectural change does.
For non-trivial findings, also evaluate:
| Dimension | What to weigh |
|---|---|
| Necessity | Does this solve a validated problem, or a hypothetical one? |
| Proportionality | Does the complexity of the fix match the complexity of the problem? |
| Evidence | What concrete evidence supports the finding over the current approach? |
| Reversibility | Can this be changed later, or is it a one-way door? |
| Codebase conventions | What do established patterns in this codebase suggest? |
Phase 4: Classify
After investigation, classify and act:
| Classification | Action |
|---|---|
| Valid bug or correctness issue (confirmed by evidence) | Fix. High priority. |
| Valid improvement (correct, tradeoffs favorable) | Implement. A correct improvement is worth implementing regardless of size — "it's a nit" is not a reason to skip a correct fix. |
| Valid — escalate (correct, but implicates a judgment call beyond current scope: product vision, risk tolerance, 1-way door) | Surface with evidence showing the issue exists and why it requires human judgment. Do not resolve autonomously. |
| Pre-existing issue unrelated to current changes (genuinely out of scope) | Decline for this scope. Note for tracking. Explain why it's out of scope. |
| Incorrect or based on misunderstanding (you have evidence) | Decline. Explain why with evidence. |
| Technically valid but tradeoffs unfavorable | Decline. Explain the tradeoffs and why. |
| Style/preference with no correctness impact | Evaluate against the ground truth chain: (1) Project tooling — if the project's linter/formatter/type-checker enforces a practice, that's the answer; run the tool, don't debate it. (2) Framework/language official docs — version-pinned from the project's lockfile. (3) Codebase consistency — tiebreaker only when tooling and docs are silent. Codebase patterns are evidence of what exists, not evidence of what's correct — especially in codebases with AI-generated code. |
| Contradicts the spec or design intent | Decline. Reference the spec. |
| Uncertain after investigation | Do not guess. Investigate further or escalate. |
The escalation boundary: Resolve autonomously when evidence alone determines the answer. Escalate when the finding's resolution requires choosing between competing values — speed vs safety, scope vs timeline, convention vs correctness, short-term pragmatism vs long-term flexibility. These are judgment calls that require human priorities, not evidence questions. Also escalate for 1-way doors (public API shape, schema changes, security boundary decisions) where the cost of getting it wrong is high and reversal is expensive.
The bar for declining: Your investigation must show the finding is factually incorrect, technically harmful, or genuinely out of scope. Never decline because:
- The issue seems minor or nitpicky (if it's correct, it's worth fixing)
- It would take effort (effort is not a reason to decline a correct finding)
- "It works as-is" (working code can still have valid improvements)
- You disagree with the style (unless codebase conventions support your position — check)
Convenience mapping for consuming skills
Consuming skills that need simple routing can use this grouping:
| Routing category | Maps to classifications above |
|---|---|
| Act | Valid bug, Valid improvement, Style/preference (when codebase supports) |
| Escalate | Valid — escalate, Uncertain |
| Decline | Pre-existing/out-of-scope, Incorrect, Tradeoffs unfavorable, Contradicts spec |
If the classification is clear and the action is obvious, proceed to Phase 6: Communicate. If the right response isn't immediately clear or the classification raises follow-on questions, use Phase 5: Resolve first.
Phase 5: Resolve
When a finding requires action beyond simple classification — the right response isn't immediately clear, or the classification raises follow-on questions — use graduated escalation.
Exhaust cheaper levels before escalating:
| Level | When | Action |
|---|---|---|
| Level 1: Re-examine existing evidence | Default first step | Re-read the source material for both sides. Check if synthesis misrepresented the evidence, or if the evidence itself is the issue. Many findings resolve here. |
| Level 2: Narrow verification | Existing evidence insufficient, specific factual question | Quick, pointed research. The question comes directly from the finding: "Is X deployed?" "Does the API require auth for this endpoint?" |
| Level 3: Deeper exploration | Finding stems from an angle the original work didn't cover | Substantive new research, still scoped to the finding area. If this reveals a much larger gap, flag it rather than expanding scope. |
Resolution types (how the resolution manifests):
| Type | When to use |
|---|---|
| Sharpen | Evidence supports a clearer claim than what's written |
| Add conditions | The truth is conditional; the finding flattened it |
| Recalibrate | Prose certainty doesn't match evidence strength |
| Acknowledge ambiguity | Evidence is genuinely unclear; forcing a resolution the evidence doesn't support |
| Re-research | Existing evidence is insufficient to resolve the finding |
Phase 6: Communicate
Match your response's certainty to your actual evidence level:
- HIGH confidence: State your conclusion directly with evidence references
- MEDIUM confidence: Hedge appropriately — "Based on my reading of X, I believe Y, though I may be missing context about Z"
- Overstating certainty on MEDIUM-confidence assessments erodes trust and can lead to incorrect decisions
When accepting: brief acknowledgment of what you're fixing and why the finding was correct.
When declining: specific evidence — code references, spec sections, tradeoff analysis, web search results, or documentation. Be direct. Do not apologize for disagreeing; explain why.
When partially accepting: explain what you're taking and what you're not, and why.
Phase 7: Structured Output
After assessing and communicating all findings, emit a Declined Findings Summary section in your response. This structured output lets consuming skills (/review-cloud, /review-local, /ship) capture declined items without re-deriving them from conversation context.
Include only findings classified as Decline (pre-existing/out-of-scope, tradeoffs unfavorable) where the finding is technically valid — i.e., the premise is correct but the fix isn't warranted in this scope. Do not include findings classified as Incorrect or Contradicts spec — those are wrong, not deferred.
| Finding | Classification | Future-relevant | Evidence |
|---|---|---|---|
| writeTracker coupling across modules | Pre-existing, out of scope | Yes | Valid encapsulation concern, not caused by this PR |
Future-relevant: Whether this item is worth tracking for future work. Set Yes when the finding identifies a real improvement that applies in a different context. Set No when the finding is purely contextual and won't apply later.
This section is consumed by:
- /review-cloud — includes future-relevant items in its completion output
- /review-local — appends future-relevant items to
review-status.json'sdeferredFindings[] - /ship — ingests from both review skills and persists to
state.json'sdeferredScope[] - Standalone callers — read the section directly from the response
When no findings are declined, omit this section.
Guardrails
Anti-deferral
Do not commit to work outside your current scope. "We'll address this in a future iteration" or "this can be revisited later" is never an acceptable resolution — you have no authority to guarantee future work will happen.
If you cannot address a finding now, decline it with evidence-backed reasoning — explain why it's incorrect, out of scope, or harmful. A clear decline with rationale is honest. A deferral is a promise you can't keep.
(Within a bounded convergence loop, deferring to the next pass in the same loop is acceptable — the loop enforces it. Deferring to unspecified future work outside the loop is not.)
Re-raised findings
When a finding you previously assessed is re-raised after your response — whether by a human pushing back or an AI re-flagging — the other party has considered your reasoning and disagreed.
Do not repeat the same rationale. Either:
- Change your position (the other party's counter-argument is convincing)
- Provide new evidence they haven't seen (a different code path, different documentation, different test result)
Repeating the same rationale a second time is not productive — it was already considered and rejected.
If your original evidence remains stronger than the new counter-argument, maintain your position. Persistence alone is not evidence — do not lower confidence because a finding is re-raised without new substantive reasoning.
Self-check before finalizing
For each finding you are declining, pressure-test:
- Did you web-search the claim, or only check the codebase?
- Would a senior engineer find your evidence convincing?
- Are you declining because the finding is wrong, or because fixing it is inconvenient?
- If the finding cited a best practice, did you verify whether it applies here?
For your overall assessment:
- Am I over-indexing (accepting everything uncritically)?
- Am I under-indexing (dismissing findings to avoid work)?
- Would someone with different experience reach a different conclusion from the same evidence?
- Verify your confidence label matches your evidence: HIGH requires cited evidence; MEDIUM requires noted uncertainty.
When evaluating AI-generated findings specifically, over-indexing (accepting invalid feedback) is the more common and more damaging failure mode — when you are the first evaluator of raw AI feedback. Weight your self-check toward rejection resistance in that context.
However, when you are a fixer responding to a structured, adjudicated review (e.g., a multi-reviewer dispatch that has already classified findings by severity), both failure modes are equally likely — the review has already filtered noise. Weight your self-check evenly. Under-indexing (skipping valid improvements because "it's just a Minor") is as damaging as over-indexing in this context.
(This asymmetry is also less pronounced when assessing human reviewer feedback, where under-indexing is equally risky.)
When assessing findings from cold readers (subagents, auditors, reviewers who didn't participate in the decision process), recalibrate severity — cold readers systematically over-severity because they lack context for why trade-offs were made. But do not discount validity: cold readers also have a fresh-perspective advantage — they catch patterns that familiarity blindness obscures. The severity may be overstated, but the issue may be real.
Over-indexing signals: Applying every suggestion without evaluating tradeoffs. Refactoring adjacent code that wasn't flagged. Adding defensive checks for impossible scenarios.
Under-indexing signals: Dismissing findings without investigating. Accepting "it works" as sufficient evidence. Declining because fixing requires effort rather than because the finding lacks merit.
Batch evaluation
When evaluating multiple findings, assess every finding before implementing any. Assessment rigor is flat — every finding gets the same investigation regardless of the reviewer's severity label. Group related findings (same code area, same theme) to catch cross-finding interactions.
After all findings are assessed and classified, implement in severity order (Critical first). Separating assessment from implementation ensures every finding gets properly evaluated — severity determines fix order, not investigation depth.
When evaluating multiple findings about the same subject, check whether earlier assessments change context for later ones — a decline on finding #2 may affect the validity of finding #5.
If two or more findings describe the same underlying issue in different terms, merge them into a single finding before evaluating — assess once, not repeatedly.
AI false positive patterns
When evaluating AI-generated code changes specifically, load references/ai-false-positive-patterns.md for 8 common patterns that AI coding agents systematically produce. These patterns help distinguish genuine improvements from AI over-engineering.