drives this directory's src/litsearch/ 's lit CLI pipeline. Core premise:Recall is not a prompt problem, it is a retrieval engineering problem.
Repeated WebSearch returns "the top few results by relevance ranking", which is neither reproducible nor able to show what was missed.
Tool location and where artifacts go (step one, don't skip)
The CLI is installed right inthis skill directory(src/litsearch/ + pyproject.toml).
All commands must be run in this directory—it is where uv can find the project:
SKILL_DIR=~/.claude/skills/lit-search # 本 SKILL.md 所在目录
cd "$SKILL_DIR" && uv run lit --help # 跑不通就先跑一次 ./install.sh
$LITSEARCH_HOME If set, use it first (people who install the tool elsewhere override with this).
If neither location finds src/litsearch/cli.py → ask the user, don't guess。
Artifacts must be written into the user's project, not the skill directory.
--runs-rootdefaults to the relative pathruns/, while the command's working directory is the skill directory— if not specified explicitly, hundreds of MB of collection results will pile up in~/.claude/skills/lit-search/runs/, where the user can neither find them nor think to clean them up.Every command must include absolute paths:
PROJ=/abs/path/to/用户的项目 # 开工前先跟用户确认这个位置
cd "$SKILL_DIR" && uv run lit harvest --topic "$PROJ/topics/<id>.yaml" \
--runs-root "$PROJ/runs" --depth standard
Protocol YAML is the same—write it under the user's project's topics/ , not in the skill directory.
This directory's examples/ has two real protocols (one biomedical, one pure CS) to copy from.
For readability, the command blocks below omit these two path parameters,but they must be supplied when actually executing。
Decide whether to use
Use: systematic reviews, the literature search section of a research proposal, challenge-competition surveys—situations that needa clear statement of recall boundaries.
(The research proposal'swritingis handled by the research-proposal skill—it consumes this skill's dois.md/references.bib, don't write the body text here.)
Don't use: the user just wants to quickly see a few representative papers. In that case WebSearch directly is faster, don't drag them through the whole pipeline.
Contract
Input: one-sentence research topic + optional time range (defaults to the last 5 years if not given). Output:
| File | Content |
|---|---|
dois.md |
Full DOI list, sectioned by quality tier, each entry with an evidence marker |
references.md |
Reference list, in the same order as dois.md consistent |
references.bib |
BibTeX(--bibtex, requires an extra fetch pass) |
PRISMA.md / coverage_report.md |
Counts and recall evidence |
Deliver everything; tiering only determines order, not inclusion or exclusion.
Three reference files, read as needed, don't read them in advance:
references/topic-template.yaml(for writing protocols),
references/jev-profile-template.yaml(required reading before using --model jev / lit triage / lit facets —threshold configuration for the judgment API),
references/evidence.md(the measured numbers behind each rule below, needed only when the user asks follow-up questions or troubleshooting).
Stage 0 | Write the protocol (this step is the model's core work)
There is no lit init command—turning a topic into a protocol requires judgment, not code. The template is at
references/topic-template.yaml, copy it to topics/<id>.yaml and fill it in following the four steps.
① Break down concept blocks. 3–5 blocks, mark which required。
The first required block becomes the anchor phrase for sources without boolean search capability (CVF/OpenReview/Zenodo). Put the most discriminative concept first—usually the disease/object/material. Putting something generic like "segmentation" will pull the entire CVPR (about 9,200 papers) back to local filtering.
② Nominate the gold standard—5–10 papers, DOIs must be verified before being written in.
gold_set is the only thing in the entire pipeline that canfalsify. So:
curl -sI -o /dev/null -w "%{http_code}\n" -L "https://doi.org/<doi>" # 200 才许写进协议
Go through doi.org, not Crossref. arXiv's 10.48550/arXiv.* is registered with DataCite,
api.crossref.org it always gets a 404—verifying with Crossref will reject genuinely existing preprints as fabricated DOIs,
while the gold standards for CS/AI topicsare almost all arXiv。
Fabricating a formally valid but nonexistent DOI costs you not "one fewer reference," butthe entire verification chain failing— you will see 8/9 recall, then spend half a day investigating a missed item that does not exist at all.
The quantity cannot be fudged. When there are fewer than 5 items, coverage_report.md will itself mark "weak evidence":
3/3 and 9/9 print exactly the same, but the former can falsify far fewer retrieval defects.
At the same time nominate 1–2out-of-windowreal papers as out_of_window_controls, to verify that date filtering really takes effect.
③ Dry-run to calibrate the order of magnitude.
uv run lit strategies --topic topics/<id>.yaml # 只看检索式,不发请求
uv run lit plan --topic topics/<id>.yaml # 各源预计命中数
| Hit count magnitude | Judgment | Action |
|---|---|---|
| > 50,000 | Concept too broad | Add a required block, or tighten the wildcards |
| 200 – 20,000 | Appropriate | Freeze |
| < 200 | Too narrow | Add synonyms; check British/American spelling variants (ischemic/ischaemic) |
This is the only window where parameter tuning is allowed. Once collection begins, changing terms will change the semantic fingerprint, invalidating the entire round.
④ Domain judgment. Topics outside biomedicine (pure CS / engineering / materials) should proactively remind the user oftwo things:
- Abstract coverage will decrease. This pipeline's abstract coverage for closed-source publishers is essentially propped up by PubMed
(measured: Elsevier 95.6%, Springer 80.4%, while these entries on Crossref have no abstracts at all).
When PubMed is unavailable, initial screening degrades to titles only.Semantic Scholar was supposed to fill this slot, but the current version has not implemented it yet
—the protocol says
semantic_scholar: enabledwill be discarded (the CLI will show a red warning). Do not claim to the user that coverage has been supplemented. - Capture-recapture will be structurally unavailable. It requires both PubMed × OpenAlex sources to be present at the same time.
After PubMed is turned off, the recall evidenceis left with only the single anchor of the gold standard—so for this kind of topic, the
gold_setmust be nominated to the full 5–10 items.
Stages 1–7 | Pipeline
After each step finishes, read the numbers aloud to the user; do not advance silently.
First let the user choose the depth—cost must be visible before execution:
uv run lit depths # 三档的规模、耗时、成本对照
| Tier | Sources | Snowballing | Scale | Time | API cost |
|---|---|---|---|---|---|
quick |
Core four sources | Not run | ≤2,000 | ~10 min | $0.1–0.3 |
standard(default) |
All sources | 1 round | 2k–8k | ~40 min | $0.3–1.2 |
systematic |
All sources | to saturation | 10k+ | 2–4 h | $3.4–10.4 |
The difference between the three levels is only inhow many sources to run and how many rounds of snowballing. The decision rules, deduplication, time window, and recall evidence
are completely identical across the three levels—downgrading reduces coverage, not rigor。
systematic must be explicitly specified and wait for user confirmation.
<run> where does it come from:lit harvest After the first line runs, it prints the run directory, and that is it.
Subsequent commands also accept <topic>/<run_id>, orgiving only the topic id from the protocol automatically takes the latest run—
the latter is the easiest. If you forget which step you ran to, use lit show <run>。
# 1 采集
uv run lit harvest --topic topics/<id>.yaml --depth standard
uv run lit proceedings <run> --topic ... # 仅 MICCAI/LNCS 类主题;语料里没有 LNCS 会自动跳过
# 2 筛选(三级放量,见下)
uv run lit screen <run> --topic ... --limit 4
uv run lit screen <run> --topic ... --limit 100
uv run lit screen <run> --topic ...
# 在 human_queue.csv 填 decision/reviewer/reason 后导回(可分批)
uv run lit adjudicate <run> --topic ... --kind screening --input human_queue.csv
# 在 boundary_cases.csv 填 decision/reviewer/reason 后导回
uv run lit adjudicate <run> --topic ... --kind date --input boundary_cases.csv
# 3 引文闭包(必须在筛选之后)
uv run lit snowball <run> --topic ... --seeds included
uv run lit screen <run> --topic ... --resume # 只补新增记录,不重判已判过的
# 4 证据与报告
uv run lit validate <run> --topic ...
uv run lit report <run> --topic ...
uv run lit verify <run> --strict
# 5 分层与交付(--ccf 可选,先问用户要目录路径)
uv run lit rank <run> --ccf CCF.pdf --included
uv run lit deliver <run> --topic ... --ccf CCF.pdf --style gbt7714 --bibtex
How the snowballing loop ends. snowball → screen --resume → snowball → …, alternating.
There are three kinds of termination, and only the first can claim closure completion:
| CLI prints | Meaning | Action |
|---|---|---|
| "saturation reached" | new inclusions in 2 consecutive rounds < 5% | stop, can say closure complete |
| "no new qualifying seeds" | natural exhaustion | stop, can say closure complete |
| "maximum rounds reached but not yet saturated" | hit the cap | cannot say closure complete—either add --max-rounds, or state explicitly in the report |
--depth quick does not run snowballing,standard only snowballs 1 round; to snowball to saturation use systematic
or explicitly --max-rounds。Do not rerun screenover the full set—that will re-judge thousands of already decided items,
doubling the cost, and may also trigger the guardrail "the number of new-round judgments is less than the previous round" and refuse to write.
Supplementing sources cannot rely on resuming the run. harvest --resume only reruns the unfinished search expressions from the previous round; itdoes not have the ability
to cover a source that has never been run. If you first ran quick and later want to add top-conference sources, use:
uv run lit harvest --topic ... --into <run> --phase stage3 --sources cvf,openreview,grey
Before scaling up, you must first smoke test, then calibrate
Do not run screening directly on tens of thousands of records. Scale up in three levels; each level costs only ten times the previous one, but reveals completely different problems:
-
--limit 4(about $0.003) verify the API contract. A fake client can test your code, but not server-side constraints. In practice, this is the step where you hitThinking mode does not support this tool_choice。 -
--limit 100(about $0.02) verify the task design. Both axes must be checked; missing either one will let problems through:Cause—a large number of
unclearwith identical reasons = applying the full-text-stage standard to the title-abstract stage; you should change thepromptrather than change the protocol (changing the protocol will alter the semantic fingerprint and block subsequent snowballing merges). A healthy queue should mainly consist of "channel disagreement" and "low confidence".Volume—if the queue share exceeds 15% you must stop and change the standard, no matter how "healthy" the cause distribution is. In one measured run of 4,000 records, the queue rate was 24.6% (973), and the cause distribution was completely normal, but 973no one will actually look at them。
lit adjudicatecan import adjudication results back, but it is meant for dozens of records; with thousands, the correct action is to come back and change the decision standard, not dump the queue on the user. Before adjudication is imported back, these records do not enterincluded, nor the deliverables. -
Scale up.
The decision API backend also has three levels:--limit 4(about $0.0002) verify server-side constraints,--limit 100
(~$0.005) validate the task design, then scale up. This step cannot be skipped—a fake client can test your code,
but it cannot test the server, nor the runtime: ten CLI tests all green with respx, but the first real run threw
Event loop is closed(httpx's connection pool is bound to the event loop that created it, while respx
intercepts at the transport layer and never opens real connections).
--limit write the results to screening_trial.json and human_queue_trial.csv(for the judgement API, write to
jev_scores_trial.jsonl),does not count as an official round—otherwise one calibration will overwrite the full results.
During calibration you open these three *_trial files, not human_queue.csv / jev_scores.jsonl。
⚠️
--limit Ntakes the first N entries of the corpus, not a random sample. prefix contains almost entirely the records that were collected first and are also the most typical, while it is always the later marginal records that blow up the human queue. Measured once: the prefix sample was 100/100 containing anchor keywords, with only 12% lacking abstracts, reporting a queue rate of 9%; while the full corpus was 22.9% and 75.7%, and re-testing with a same-distribution random sample gave a true value of 18%, exceeding the red line.after running
lit proceedingsor after snowballing the gap is largest—the records appended in those two steps all sit at the tail of the corpus. Before calibration, check the sample composition(whether the keyword hit rate and the proportion lacking abstracts are comparable to the full corpus); if not comparable, copy the run to a temporary directory, replacecorpus.jsonlwith random N entries and rerun, don't modify the original run.
Order that cannot be skipped
Screening must come before snowballing. Measured: starting from 13,920 unfiltered seeds, the backward candidates are 134,317;
yet only a few hundred will ultimately be included. Skipping screening to snowball = amplifying noise by an order of magnitude, and paying ten times the screening cost for it.
Even when the user is pushing, don't switch to --seeds in_window—first explain the cost clearly and let them decide.
Snowballing and screening must alternate. Seeds are "already included"; without screening there are no seeds. Measured saturation curve 34.7% → 4.1% → 0.4%, while the unique contribution of snowball accounts for 92%— this is direct evidence of how much pure keyword retrieval would miss.
Cost must be visible before execution. lit screen --dry-run gives token count and dollar amount.
When there are no credentials it is a rough character estimateclearly labelled as such. Don't skip this step on the user's behalf.
Tiering and delivery
Only label and rank, never filter. Thresholds turn "never evaluated" and "low quality" into the same action— measured with a cut at 4.0, 1,487 entries were reduced to only 390, of which the 340 deleted werehaving no metricsrather than having low metrics.
Tiering takesthe best of all available yardsticks:
| Tier | Criterion |
|---|---|
| S Top conferences and journals | CCF-A, or paper citations entering the top 1% of the same field |
| A High level | CCF-B, or journal metric ≥ 8.0, or citations in the top 10% |
| B Quality | CCF-C, or journal metric 4.0–8.0 |
| C Latest preprints | preprints / repositories, ≤ 18 months |
| D Conferences (unrated) | not in the CCF list—unrated, not low-tier |
| E Other journals | Journal metric < 4.0 |
| F Others | No basis for any evaluation—not evaluated, not low quality |
Four things that must be made clear:
- CCF and impact metrics do not measure the same thing; take the better one. Measured: Medical Image Analysis is CCF-C, while its OpenAlex metric of 10.14 is the highest in the entire corpus.
- OpenAlex
2yr_mean_citednessIt is not JIF. The JIF denominator counts only citable items, while the OpenAlex denominator counts all items; journals publishing large numbers of conference abstract supplements are systematically underestimated by three to five times (measured: Stroke 1.81, Neurology 1.11).Thresholds cannot be carried across scales. - Leaderboard data is provided by users and is not distributed with the repository. The CCF catalog is a public document but still goes through the
--ccfslot; JCR / CAS partitions are commercially licensed data,and must absolutely not be packaged into the repository。 - without a catalog, do not guess the tier. Conferences not in the catalog are marked "unrated". A fabricated CCF-A looks exactly the same as a real one—this is the most dangerous part.
Citation formatting requires no key:lit deliver It uses doi.org content negotiation, supporting
apa / ieee / gbt7714 / nature / ama / bibtex。
(journal matching uses ISSN rather than journal name, and style names are validated before sending the request—these two are lit rank / lit deliver
internal guardrails; you have no corresponding operation. To learn why, see references/evidence.md。)
Credentials
are read from .env or environment variables (real environment variables take precedence). The consequences of missing each one differ, and must be stated clearly:
| Missing | Consequence |
|---|---|
LITSEARCH_LLM_API_KEY |
not fatal—use --model host to go through the host model, zero configuration |
OPENALEX_API |
Mid-collection QuotaExhausted; capture-recapture cannot be computed, and the recall evidence is left with only a single gold-standard point |
CONTACT_EMAIL |
lit fulltext refuses to run outright—both Unpaywall and Crossref require declaring identity |
Screening backends: three paths
API backend(default): any OpenAI-compatible endpoint. Three environment variables
LITSEARCH_LLM_MODEL / _BASE_URL / _API_KEY. Built-in measured pricing for DeepSeek;
pricing for other vendors is provided by LITSEARCH_PRICING —when there is no pricing, --dry-run will truthfully report
"cannot estimate", and will not report a fake $0.00。
Host backend(--model host): calls the local claude -p,requires no key at all.
It runs right after cloning—this is the zero-configuration entry point for community users. Three things must be made clear:
- each call has about 20,000 tokens of framework overhead, so batches are automatically enlarged to 60 items
- Itis not free: measured, 60 records cost about $0.30 in API-equivalent cost (haiku; with opus it is $1.44). Subscribers do not pay extra, but it consumes quota—do not call it "$0"
--host-modeldefaults to haiku. For scales of tens of thousands, switch to the API backend
Decision API backend(--model jev): TypeSafe Jev does not generate text; for each inclusion/exclusion criterion
it returns a calibrated probability. $0.042 per million input tokens, output free—an order of magnitude cheaper than generative LLMs.
But it has no general default threshold; the configuration file is required. Measured with the same set of thresholds, on the pediatric project it pushed the
manual queue down from 18.3% to 11.2% (AUC 0.987), while on the ISLES project it was actually worse than the existing LLM
(8.5% vs baseline 7.2%). The failure cause is traceable:X1"Only performs hemorrhagic stroke...and noischemic
stroke data" — for criteria like this with multi-hop indirectness + implicit negation, AUC alone is only 0.675.
So before scaling up, it must first pass admission; all three thresholds are indispensable:
- Gold-standard seeds 100% retained (being automatically excluded directly returns 1, no judgment needed from you)
- Manual queue ≤15%
- Not worse than the project's existing LLM baseline—this one is the easiest to overlook, and it is the one ISLES failed on
lit screen-eval <run> --topic ... --reference-round <人工裁决过的轮次> \
--candidate-scores jev_scores.jsonl --score-keys I1,I2
The report will give the numbers for gold standard (human adjudication) and silver standard (model judgment) separately, and label the silver standard as a ceiling— measured on one project, about 21% of silver-standard inclusions were overturned by humans. Each rate comes with a Wilson interval.
Scores fall in jev_scores.jsonl, changing the threshold only reroutes, it does not spend money again.
OpenAlex has a daily quota. lit deliver By default reuse ranked.jsonl, do not refetch metadata;
to refresh, add --refresh. When the quota is exhausted, QuotaExhausted will include the reset seconds—slowing down is ineffective, do not retry.
Levels of recall evidence (clarify before reading the report)
- Gold-standard recallmeasures "how much of the literature you already know has been retrieved"—it has nothing to say aboutunknown unknowns and the gold standard itself is biased toward literature already covered by retrieval habits (circularity is the inherent upper limit of this metric).
- Capture-recapture estimation(
lit validate, PubMed×OpenAlex dual-source) is the statistical statement about overall missed detection;snowballing saturation curveis independent behavioral evidence. Read the three together: the gold standard is lower-bound validation, recapture and saturation are the answer to "how much is still unfound".
Update-type review:lit diff
retrieval is a snapshot. Moving the window forward or protocol evolution both requirereopening the run(fingerprint discipline, changing the window = new protocol),
then use lit diff <旧run> <新run> for reconciliation: the list of records still present/new/disappeared, retraction emergence in the new corpus, and you can --carry-adjudications carry over the old run'smanual final review(machine triage
is not carried over) by key into lit adjudicate a CSV that can be imported directly—updating an SLR does not require
screening from scratch.--output Also save a human-readable difference report.
learnings: review before retry
Before changing decision logic, thresholds, collection behavior, or artifact contracts, scan learnings/—
inside is a list of incidents bought with repeated real-world measurements (numbers, costs, the judgment at the time), not style advice.
New incidents (rework, silent under-recall, docs and implementation disagreeing) get appended afterward as an entry, named by date.
Discipline for reporting numbers
- "The parts not executed this time" must be read out to the user.
lit reportwill compute it and write it intocoverage_report.mdthe first section of , and the terminal will also list it in yellow text.Do not skip this section to talk about the inclusion count: "Ran but didn't succeed" was governed by a partial alert, while "never ran at all" previously left no trace anywhere in the entire set of artifacts— in a measured delivery, the manifest declared 4 sources but only 2 were actually accessed, and at the time not a single artifact mentioned this. - The partial/failed sources must be named. Its recall number is a lower bound, not the actual value.
- Passing the title-and-abstract initial screening is also not final inclusion. Before screening, the number of records in the window is not the number of relevant papers;
after initial screening is complete,
includeit is still only a candidate report entering full-text eligibility review.lit reportWithout full-text retrieval, exclusion reasons, and a final eligibility record, it will stop at Eligibility input, and you must not fill in a final outcome in your wording either. - A measured zero must be stated. "Among 9,200 CVPR/ICCV/WACV 2021–2026 papers, there was not a single one" is a valuable conclusion, completely different from "I think there isn't any there."
- Gold-standard missed detections must be attributed one by oneto a specific search expression or source defect; you cannot vaguely say "recall is good."
- Chapman is used only for two-source overlap diagnostics. PubMed and OpenAlex are not independent; positive correlation makes the union coverage of the two sources optimistic; it did not verify topical relevance, nor is it the recall rate of a multi-source pipeline.
Three hard rules
① A 200 does not mean you got what you wanted. An enumerating source that returns "200 but parses 0 records" must be reported as a failure; if content negotiation brings back HTML, it must be judged a failure. In measured testing both of these actually happened, and the second recurrence was caused by me blocking only "bad samples I had seen before" (JSON errors). Validation must be written as "confirm it is what I want," not "exclude bad samples I have seen."
② A failed run must not overwrite a successful run. If the new round's judged count is less than the previous round's = refuse to write, exit nonzero. The information needed for resuming from a checkpoint (such as channel number) must be saved on the first run—afterward it either cannot be reconstructed, or can only be guessed, and a guessed value looks the same as a real one.
When you hit this guardrail (exit code 1 + "do not write a new round"):do not retry the full run. First look at failure samples to locate the cause (quota exhausted / network / endpoint changes), and after fixing it use
screen --resumeto fill in only the missing judgments.
③ Do not re-collect after rules evolve. lit reclassify Use the current rules to recompute from records*.jsonl deduplication
and the decision window, without sending a single request. After a source's parsing defect is fixed, use
lit harvest --into <run> --phase fix --labels <那几条> for a targeted backfill run.
Collection is an expensive one-time action; judging is not.
How to account for the two manual queues
human_queue.csv(decision disagreement / low confidence) and boundary_cases.csv(insufficient date precision)
can now both flow back through lit adjudicate , preserving the adjudicator, time, reason, and original judgment.
So after running, two things must be done, and neither can be done on the user's behalf:
- Report the two files'paths and countsto the user, and explain the
decisionallowed values; do not adjudicate on the user's behalf.--kind screeningacceptsinclude/exclude(does not accept remaining asunclear);--kind dateacceptsin_window/out_of_window, andonly accepts boundary / undated records ——adjudication is for resolving uncertainty, not for overriding existing decisions. If a confirmed date was judged wrong, fix the date evidence ordate_priorityand rerunlit reclassify。 - the user fills in
reviewer、reasonthen run the correspondinglit adjudicate --kind .... After importing back, report the remaining queue and runlit reportandlit verify. When the queue is unrealistically large (>15%), calibrate the standard first.
--kind date after moving boundary records into the strict window, these recordshave not yet been screened,
lit verify will report screening-corpus-mismatch. This is the correct reminder: rerun
lit screen <run> --topic ... --resume before producing the report again.
Push to Zotero
lit report generate zotero_dois.txt. Add each one by DOI via the Zotero MCP (tool names depend on the installed MCP—the one matching this repository is zotero_add_item, just pass the DOI).
Before pushing, confirm the collection name—this is a write operation to an external system, do not execute it by default.
This list is thefull-text candidate setthat passed title/abstract initial screening, not the final included studies. Candidates without a DOI are not in the
list; do not fabricate identifiers out of thin air; after pushing, tell the user about this.
Artifacts
runs/<topic>/<timestamp>/:
| Files | Content |
|---|---|
raw/<阶段>/<attempt-id>/ |
The gzip raw per-page response of each attempt; reusing old pages is not allowed when fixing and rerunning |
records*.jsonl |
Records per source, split into files by collection stage, append-only and never rewritten |
corpus.jsonl |
The deduplicated canonical corpus (a derived artifact, recomputable at any time) |
manifest.json |
Protocol fingerprint + the outcome and status of each search query |
boundary_cases.csv |
Insufficient date precision, and merging of titles in borderline regions (similarity 95–97.5, merge uncertainty)—requires manual adjudication |
human_queue.csv |
Dual-channel disagreement or low confidence, requires manual adjudication;lit triage will reorder it by most likely inclusion and attach adjudicator clues |
fulltext.jsonl / institutional_access.csv |
lit fulltext resolution of legitimate full-text locations for --included used after screening); records for which open full text cannot be obtained go into the institutional subscription list |
screening_trial.json / human_queue_trial.csv |
--limit trial-run rounds and queue (look at these two during calibration, not human_queue.csv); does not occupy an official round number |
jev_scores.jsonl |
Per-item scores from the adjudication API (--model jev / lit triage). Changing the threshold only requires rerouting, not re-scoring |
jev_scores_trial.jsonl |
lit screen --model jev --limit N the trial-run scores; when calibrating the threshold, use them as --candidate-scores input |
screen_eval_*.md / *_disagreements.csv |
lit screen-eval 's evaluation report and disagreement list (once filled in, can be directly lit adjudicate) |
facets.jsonl / facets.csv |
lit facets annotations: external validation, multicenter, public datasets, etc. Only stated/not_stated,there is no no |
PRISMA.md |
Counts for each stage (generation is refused if inconsistent) |
coverage_report.md |
The part not executed this time, strict-window gold-standard recall, out-of-window leakage, unique contribution, two-source overlap diagnostics, saturation curve |
artifacts.json |
Derived artifact content hash and input dependency hash; by lit verify check for expiration |
ranked.jsonl / ranked.csv |
Stratified annotation (ranked.jsonl is a metadata cache) |
tier_report.md |
Stratified composition and level-by-level match rate |
dois.md / references.md / references.bib |
Deliverables. OpenAlex marks retracted entries in a separate "⚠️ Retracted" section and flags them inline; during filtering, retracted records are forcibly escalated to the manual queue (machines may not definitively judge retracted literature, but manual adjudication may) |