A model that reviews code for defects also invents them: it misreads a guard
ten lines up, a loop bound, or a lock the caller holds. This skill splits the
review into two roles, as the open-source reviewer bluesky-social/roast does.
A finder (you, the code-review skill, any reviewer) writes each finding
with a verbatim quote of the code it is about. A verifier from a different
model family then reads that code and confirms or refutes the claim.
The verifier is Jev, a decision model from TypeSafe. It answers typed questions (yes/no, multiple choice, ordered scale) with a probability for every answer in about 0.3 s. Jev is not a Claude model, so when Claude finds and Jev verifies, the verifier is less likely to share the finder's blind spots. Below, P(x) means the probability Jev gives to answer x.
scripts/verify.py does the following for each finding:
- Grounding. The quote must occur in the file, compared with runs of whitespace collapsed. A finding with no quote, or a quote that is not in the file, is dropped before Jev sees it.
- The window. The verifier sees a window of the file: the top-level
block (function, class, statement) that contains the quote, with at least
12 lines on each side, capped at 160 lines. Lines are numbered and the
quoted ones are marked
>. With--diff, the file's diff hunks go along too. - Three questions to Jev, one call per finding, in parallel:
- verdict:
confirmed,refutedorinsufficient_context - severity if the defect is real, on the scale
info,low,medium,high,critical - when the finding proposes a fix, whether the fix works
- verdict:
- Thresholds. P(refuted) of at least 0.7 drops the finding
(
--refute-at). P(confirmed) of at least 0.7 confirms it (--confirm-at). Anything else isuncertain. Only a refutation removes a finding; a verifier that is unsure keeps it.
When NOT to use this skill
| Situation | Use instead |
|---|---|
| Nobody has looked for defects yet | code-review (or any reviewer) first; this skill only judges findings that exist |
| Checking that a README or docstring still matches the code | verifying-claims |
| An adversarial read of a whole deliverable (prose, analysis, a PR's design) | challenging |
| A routing, triage or yes/no question that is not about a code finding | deciding-with-confidence, or the Muninn decide tool directly |
| Building a deterministic check that must be able to go red | gating |
| The claim is about runtime behaviour the code cannot show (performance, a flaky test, production data) | run it; a verifier reading code can say insufficient_context at best |
Stop the run when most findings come back uncertain with reason
needs_more_context: the evidence for those claims lives outside their
windows (in a caller, another file, a config). Verify those findings by
reading the code yourself.
Procedure
Throughout, V=/mnt/skills/user/verifying-review-findings/scripts/verify.py.
-
Write the findings as a JSON list.
file(relative to the repository root) andcode_quoteare required. Copy the quote character for character from the file, one to six lines; a paraphrase is dropped.[{"id": "1", "file": "src/util.py", "line": 6, "description": "Off-by-one: the loop skips the last element.", "code_quote": "for i in range(len(items) - 1):", "fix": "for i in range(len(items)):", "severity": "high"}]The other fields are optional.
iddefaults to the item's position.linepicks between several matches of the same quote.descriptionis the claim Jev judges.fixadds the fix question.severityis the finder's own estimate and is passed to Jev as a claim. Findings in theReportFindingsshape (summaryandfailure_scenarioinstead ofdescription) are accepted too. When you are the finder, emit this shape directly. -
Pick the path.
--transport auto, the default, uses the first ofjevandgatewaywhose credentials are set.Path Needs How Jev direct ( jev)TYPESAFE_API_KEY, a TypeSafe API keyrun(step 3a)Jev via Cloudflare Workers AI ( gateway)CF_ACCOUNT_IDandCF_API_TOKEN, optionallyCF_GATEWAY_IDrun; pass--transport gatewaywhen both paths have credentialsMuninn decidetool, which calls Jevthe Muninn MCP connector in the session emit, thenapply(step 3b)Claude Haiku via deciding-with-confidencethat skill installed next to this one run --transport decider; the verifier is then also Claude -
Run it.
a. With credentials in the environment:
python3 $V run findings.json --repo . --format md # default output is JSON git diff main... > /tmp/d.diff && python3 $V run findings.json --diff /tmp/d.diff python3 $V run findings.json --ref HEAD~1 # read files at a git refb. Through the Muninn
decidetool:python3 $V emit findings.json --out batch.jsonemitsaves the grounded findings tobatch.jsonand prints the Jev requests, one per surviving finding, each as{"id", "state", "questions"}. Calldecideonce per request, passing itsstateandquestionsunchanged; the calls are independent, so send them together. Save the replies inanswers.jsonas an object that maps each finding'sidto the full textdecidereturned, then:python3 $V apply batch.json answers.json --format md -
Read the report. The top level has
complete(false when any finding could not be verified),countsper status, andgate. Each finding keeps its input fields and adds:status:confirmed,uncertain,dropped, orunverified(Jev was not reached or replied in an unexpected shape)reason: why it is not confirmed. One ofmissing_code_quote,ungrounded_code_quote,file_not_found,refuted_by_verifier,split_verdict,needs_more_context(insufficient_contextwas the largest verdict),no_answer, orverifier_error: ...verdict: the three verdict probabilitiesseverity:probabilitiesper level, andexpected, the level nearest the probability-weighted averagefix_ok: the probability that the fix worksclaimed_severity: the finder's ownseverityfrom the inputfingerprint: a hash of the file and the whitespace-collapsed quote. Use it to match a finding across runs, never to suppress one: two different claims about the same lines share a fingerprint.
-
Act on it.
- Report
confirmedfindings as findings. - Report
uncertainones separately and read their code yourself; the verifier's mistakes cluster there. - Leave
droppedfindings out, but give their count and reasons so a reader can recover one that was refuted wrongly. - When a confirmed finding has
fix_okunder 0.5, rewrite the fix before proposing it.
- Report
-
Gate a commit or CI step with
--fail-on LEVEL(any severity level). Exit codes:Code Meaning 0 every finding verified, and no gate tripped 1 a confirmed finding has P(severity at least LEVEL) of at least --severity-at(default 0.5)2 bad input, or no transport available 3 some finding is unverified: the run is incomplete, not cleanA pre-commit hook that should not block on an outage can let the commit through on 3, as long as the hook prints the report so the gap is visible:
python3 $V run staged.json --fail-on high --format md; rc=$? [ $rc -eq 3 ] && echo "verify: incomplete run, commit allowed" >&2 && rc=0 exit $rc
Earned exceptions
| Default | Change it when | Keep it when |
|---|---|---|
--refute-at 0.7 |
a missed defect costs more than reading a false one (security review, release gate): raise it toward 0.9 so fewer findings are dropped | the review is advisory, and noise is what makes people stop reading |
| The gate counts only confirmed findings | the gate guards something irreversible: lower --confirm-at so more findings count as confirmed |
uncertain findings get a human look before merge |
| One window per finding | the claim depends on another file (a caller, a type, config) | the evidence is in the same top-level block, which covers most of what reviewers flag |
Common failure modes
verify: no Jev transport, exit 2 → neitherTYPESAFE_API_KEYnor the Cloudflare pair is set. Use theemit/applypath, or--transport decider.- Every finding
droppedwithungrounded_code_quote→ the finder quoted from memory or reformatted the code, or--repopoints at the wrong checkout. Rungrep -Fwith one quote against the file to tell which. file_not_foundfor files that exist →fileis resolved against--repo(default: the current directory). A path that leaves the repository is treated as missing on purpose.unverifiedwithverifier_error: HTTP 4xx→ check the error text. When it names the input length, the window was too long for Jev's default context: lowerMAX_LINES(the 160-line cap, a constant at the top ofverify.py). The report sayscomplete: falseuntil those findings are re-run.unverifiedwithno_answerafterapply→answers.jsonhas no entry for that finding'sid. Ids are strings:"1", not1.- A high
severity.expectedon a dropped finding → severity answers "how bad if real". Read it only on findings that survive.
Verification
python3 -m pytest /mnt/skills/user/verifying-review-findings/tests -q # offline, no network
python3 $V eval /mnt/skills/user/verifying-review-findings/assets/eval.jsonl
eval runs 36 labelled findings (16 real defects, 16 false findings, 4
quotes not in the file) and prints the counts. Grounding is deterministic,
so ungrounded_dropped must be 4/4. Jev's part gave real_kept 16/16 and
false_dropped 11/16 with jev-1.14.0 on 2026-11-07. A lower count means the
Jev version or the window settings changed; compare per item with
--out results.jsonl.
A bad success is a report with complete: true in which every finding is
confirmed near 1.0, including ones you know are false: Jev is judging the
description without the code. Check that state.code in the requests emit
prints holds the numbered window.
The eval's per-item results, the threshold sweep and what it does not cover
are in references/method.md.
Diagnosed failures
- 2026-11-07, eval item
ts-04: a finding claimed a race condition in synchronous single-threaded JavaScript. Jev confirmed it at 0.89, the only false finding confirmed. No threshold catches a confident error like this. Read the code behind any concurrency claim about single-threaded code yourself. - 2026-11-07, eval items
py-05andsh-03: the evidence that refutes each claim was in the same file but in a different function, outside the window. Neither was dropped.sh-03came backuncertainwithneeds_more_context(P(insufficient_context) 0.82).py-05came backuncertainwithsplit_verdict(P(confirmed) 0.55, P(insufficient_context) 0.40). Both outcomes are correct: the finding stays in the report, flagged for a human to read.