SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsalterlab-citation-verifier

alterlab-citation-verifier

Coding & development

View repository

Updated 10-15 19:56

About this skill

Citation Verifier — Existence-Verify a Bibliography Against Public Scholarly APIs

The headline existence-verification skill: given a bibliography in any common form, it proves entry-by-entry whether each reference actually exists by querying public scholarly APIs, then maps each result to the canonical AlterLab citation-hallucination taxonomy. It is the deterministic, network-grounded companion to the LLM-driven integrity_verification_agent — where that agent uses WebSearch + judgment, this skill uses authoritative API records and a reproducible Python script, so every verdict rests on retrieved records and the same input and source responses yield the same verdicts.

Verify references with the script, not from memory. A model checking citations against what it "remembers" shares the training data that produced the hallucination in the first place, so a fabricated reference that feels right passes undetected; only an external record settles existence.

Quick Start

Verify the citations in references.bib
Check whether these DOIs resolve to the papers I cited
Audit my bibliography for fabricated / hallucinated references
Does this reference list contain any fake citations or retractions?

→ Run scripts/verify_citations.py over the bibliography, read the JSON, then present a verdict table grouped by severity. State the offline/degraded status explicitly if the network or a source was unavailable.

When to Use This Skill

  • "Verify / check / audit my citations or references exist"
  • "Did the AI hallucinate any of these references?" / "Is this DOI fake?"
  • "Do these DOIs resolve to the papers I cited?"
  • "Check this bibliography for retractions"
  • "Does this source actually support the sentence I cite it for?" (SH, abstract level)
  • A reproducible, scriptable existence check with an explicit offline mode

Does NOT Trigger

Scenario Use Instead
Writing or revising the paper itself (its citation-check mode formats citations) alterlab-paper-writer
Finding papers, extracting metadata, or generating BibTeX for new sources alterlab-citation-mgmt
Dead or redirected hyperlinks (link rot) in docs or a reference list alterlab-link-health
Full pre-/post-review integrity gate (citation context, data, originality) alterlab-research-pipeline
Holistic peer review of a manuscript with an editorial decision alterlab-paper-reviewer

Grading source quality or predatory venues belongs to alterlab-deep-research (source_verification_agent). For a whole-manuscript audit in Claude Code where every flag should be re-checked by independent agents, offer the packaged /alterlab-workflows:citation-audit workflow (alterlab-research-workflows, roughly 10–40 agents), which runs this skill's checks at scale.

What This Does

For each bibliography entry the script:

  1. Parses the input (auto-detects BibTeX / a bare DOI-or-arXiv list / free-form references, including one-reference-per-line pastes), extracting title, authors, year, venue, DOI, and arXiv ID. A full reference that merely contains a DOI stays free-form, so its title and authors are kept for the Identifier Hijacking check.
  2. Resolves identifiers — looks up the cited DOI/arXiv ID directly when present, and asks the doi.org Handle API whether a DOI is registered with any agency.
  3. Searches by title + first author + year as a fallback across the sources.
  4. Fuzzy-matches the cited title (difflib SequenceMatcher ratio, default threshold 0.70; a main title that is near-identical to a record stored without its subtitle also counts) and computes author-surname overlap. Among the returned records it keeps the one that matches title, then authors, then year — so a same-titled paper by other authors or another edition does not win.
  5. Flags retractions on the matched work: Crossref updated-by / update-to notices of type retraction, withdrawal, or removal (from the publisher or, with source: retraction-watch, the Retraction Watch database that Crossref serves in its REST API since 2025) and OpenAlex is_retracted. Expressions of concern get a separate flag.
  6. Emits a verdict per entry mapped to the taxonomy below, with a source_status map showing which sources answered, plus a repo-level summary.verdict (PASS / PASS_WITH_CONDITIONS / FAIL / UNVERIFIED).

Sources (no key required; current as of 2026-09)

Source Endpoint Used for Access notes
Crossref api.crossref.org/works DOI resolution, bibliographic search, retraction notices mailto routes to the polite pool (10 req/s, 3 concurrent)
OpenAlex api.openalex.org/works DOI lookup + title search, is_retracted mailto is ignored since Feb 2026. DOI lookups are free; searches draw on a daily budget shared per IP without a key (a free key gives 10× — set OPENALEX_API_KEY)
Semantic Scholar api.semanticscholar.org/graph/v1 DOI/arXiv resolution, title search Keyless pool is shared and often returns 429; S2_API_KEY gives a dedicated 1 req/s
arXiv export.arxiv.org/api/query arXiv ID resolution, preprint title search ≥ 3 s between calls (the script throttles)
doi.org Handle API doi.org/api/handles/<doi> Is the DOI registered with any agency (Crossref, DataCite, mEDRA…)? Authoritative for DOI existence

Keys are optional, read from the environment, sent as headers, and never written to the report (the report only records whether one was present). A source that errors (rate limit, 5xx) is recorded in source_status and is never counted as evidence that a work does not exist.

This skill answers "does the cited work exist, and does its identifier point to it?" From API metadata alone it cannot establish Semantic Hallucination (does the source support the claim?); scripts/claim_faithfulness.py gives an abstract-level SH triage (see below), and full-text claim verification is the research pipeline's claim_verification_protocol (Phase E).

Verdict Taxonomy (mirrors the canonical Five-Type Taxonomy)

Identical codes and definitions to alterlab-research-pipeline/agents/integrity_verification_agent.md (GPTZero × NeurIPS 2025; Ansari, 2026). Severity feeds the same SERIOUS / MEDIUM / MINOR scale used in the Integrity Report schema.

Code Name Severity Script trigger
verified — (exists, matches) NONE Title ratio >= threshold AND author overlap OK AND year consistent in >=1 authoritative source
TF Total Fabrication SERIOUS Cited DOI is not registered at doi.org, or cited arXiv ID does not exist on arXiv, and no close title match exists; OR (no identifier) no record matches title + authors while Crossref and OpenAlex or Semantic Scholar answered
PAC Partial Attribute Corruption MEDIUM Entry found but >=1 metadata field disagrees (year, author overlap < 50%, a cited identifier that does not resolve to it, or a garbled title whose closest record shares the authors — possible mashup)
IH Identifier Hijacking SERIOUS Cited DOI/arXiv ID resolved (method=id) but the resolved record's title is unrelated (ratio < threshold)
PH Placeholder Hallucination SERIOUS Unresolved template/placeholder ([CITATION NEEDED], \cite{}, et al., YYYY, TODO, forthcoming, in press) — caught pre-network
SH Semantic Hallucination SERIOUS Entry resolves but does not support its claim — advisory only here; asserted via claim_faithfulness.py / Phase E
unverified — (could not check) MEDIUM Offline, the sources needed for a decision errored, or a registered DOI that no index returned. Never treated as passing.

A RETRACTED flag is attached (and severity bumped to SERIOUS) whenever Crossref or OpenAlex marks the matched work as retracted, independent of the existence verdict; an unrelated closest hit never lends its flag to the cited entry.

TF is a serious accusation, so the script only asserts it on authoritative evidence. When the deciding sources were rate-limited or down, the entry is unverified with the reason and source_status, and it goes to the fallback below.

Repo-level verdict

  • PASS — every entry verified, no SERIOUS/MEDIUM flags.
  • PASS_WITH_CONDITIONS — only PAC / MEDIUM metadata issues (fixable), and every entry was checked.
  • FAIL — any SERIOUS verdict (TF / IH / PH / retraction).
  • UNVERIFIED — nothing SERIOUS was found but at least one entry could not be checked (e.g. an offline run or rate-limited sources). This is not a pass — re-run with network access or an API key, or resolve those entries manually.

Pipeline (how to run it)

1. Locate or capture the bibliography

Accept any of: a .bib file, a .txt list of DOIs/arXiv IDs, a pasted reference list, or inline text. The script auto-detects the format; override with --format bibtex|doi|freeform if detection is wrong.

2. Run the verifier

uv run python skills/core/alterlab-citation-verifier/scripts/verify_citations.py \
    path/to/references.bib \
    --mailto <contact-email> \
    --threshold 0.70 \
    --out citation_report.json
  • path/to/references.bib may also be - (stdin) or inline text.
  • --threshold tunes the fuzzy title-match ratio (0..1; default 0.70).
  • --offline skips the network and emits unverified verdicts deliberately.
  • Omit --out to print the JSON report to stdout.
  • Optional: export OPENALEX_API_KEY=… (free key from openalex.org/settings/api) and/or S2_API_KEY=… before large bibliographies — keyless OpenAlex searches share a small per-IP daily budget, and the keyless Semantic Scholar pool is busy.

The script uses requests if installed, else the Python stdlib (urllib), so it runs with no extra dependencies in a bare uv environment. Expect roughly 3–10 s per entry: it spaces arXiv calls 3 s apart and backs off on rate limits.

3. Read the JSON and report

Parse summary.verdict and the per-entry verdict codes. Present:

  1. The headline verdict and counts (verdict_counts, severity_counts).
  2. A table of every non-verified entry with its code, severity, and detail.
  3. For each TF / IH / PH: quote the cited entry and explain the evidence (e.g. "DOI 10.x resolved to an unrelated paper titled '…'", "DOI not registered at doi.org").
  4. Any RETRACTED / EXPRESSION_OF_CONCERN flags, prominently.
  5. For unverified entries: say which sources failed (source_status), relay the manual_instructions and any summary.notes (for example, an OpenAlex HTTP 429 means the keyless per-IP budget is spent — a free OPENALEX_API_KEY fixes it), and run the fallback below.

4. Route fixes

  • TF / PH → the reference must be removed or replaced; it does not exist.
  • IH → the DOI/arXiv ID is wrong; find and substitute the correct identifier.
  • PAC → correct the specific metadata field(s) named in detail; a possible mashup needs the real source identified before it can stay.
  • RETRACTED → flag to the author; cite the retraction notice or drop the source.

Graceful Degradation and the Fallback

Failures are never swallowed into a pass:

  • A DNS/connection failure raises NetworkUnavailable; the entry becomes unverified with a populated manual_instructions field.
  • A rate-limited or failing source is recorded in source_status; if the sources needed for a decision did not answer, the entry is unverified, not TF.
  • --offline forces every networked entry to unverified up front (placeholders are still caught locally as PH).
  • The repo-level verdict becomes UNVERIFIED whenever an entry is unverified and nothing SERIOUS was found.

For each unverified entry, re-run with connectivity or a key if possible; otherwise look it up with the plugin's crossref / openalex MCP tools when they are available (setup: mcp_setup.md under skills/core/references), then fall back to WebSearch with three distinct queries (exact title in quotes; title + first author; first author + venue + year) and, for a DOI, open https://doi.org/<DOI>. If a record is found, report the verdict it supports and say it came from the fallback. If nothing is found, classify the entry as TF (NOT_FOUND). There is no "difficult to verify" outcome: the Lin et al. mashup this skill was built around slipped through three integrity rounds precisely because it was parked in that gray zone and never searched.

Claim Faithfulness (SH) — scripts/claim_faithfulness.py

Existence is half of the gate; a real paper cited for something it never said (the "Frankenstein" pattern) passes every existence check. For (claim, DOI) pairs, claim_faithfulness.py fetches the cited abstract (Crossref, then OpenAlex) and returns support / contradict / unsupported:

uv run python skills/core/alterlab-citation-verifier/scripts/claim_faithfulness.py \
    --claim "<the sentence as written>" --doi <doi>
uv run python .../claim_faithfulness.py --input pairs.json --json        # batch
uv run python .../claim_faithfulness.py --input pairs.json --tier llm --json
  • The default heuristic tier is lexical and deliberately abstains (unsupported) rather than guessing support; it cannot see role reversals or wrong numbers. The llm tier asks the model set by ALTERLAB_MODEL (see shared/model_env.md) and falls back to the heuristic, flagged, if the claude CLI is unavailable.
  • It only sees the abstract (abstract_only: true). unsupported means the abstract does not establish the claim — non-coverage, not refutation. Report contradict as SH; send unsupported claims that matter to full-text checking.
  • Always report existence and faithfulness separately: "the citation is real" and "the citation supports this sentence" are different findings.

Output Shape (excerpt)

{
  "tool": "alterlab-citation-verifier/verify_citations.py",
  "version": "1.1.0",
  "summary": {
    "total": 2,
    "verdict": "FAIL",
    "verdict_counts": {"verified": 1, "TF": 1, "PAC": 0, "IH": 0, "PH": 0, "SH": 0, "unverified": 0},
    "severity_counts": {"SERIOUS": 1, "MEDIUM": 0, "MINOR": 0},
    "citation_integrity_score": 0.5,
    "fabrication_risk_score": 0.5,
    "retracted": 0,
    "source_errors": {"openalex": 1},
    "notes": ["OpenAlex rate limit: the keyless daily budget shared by this IP is spent ..."]
  },
  "entries": [
    {"ref_id": "walters2023", "verdict": "verified", "severity": "NONE",
     "title_ratio": 1.0, "author_overlap": 1.0, "matches": [{"source": "crossref"}],
     "source_status": {"crossref": "record", "openalex": "record", "semanticscholar": "record", "arxiv": "no_record"}},
    {"ref_id": "ghostpaper2021", "verdict": "TF", "severity": "SERIOUS",
     "detail": "DOI 10.xxxx/... is not registered with any DOI agency (doi.org Handle API)..."}
  ]
}

citation_integrity_score and fabrication_risk_score (both 0..1) align with the Integrity Report schema fields of the same name, so the report can feed alterlab-research-pipeline's integrity gate directly.

Self-Check Before Reporting

  • Did the run reach the network, and which sources answered? If entries are unverified, name the failing sources from source_status; do not imply a pass.
  • Are there any RETRACTED flags? Surface them even on otherwise-verified entries.
  • Did any entry score IH? Confirm the detail shows an id-resolved mismatch, not a loose title-search coincidence (the script enforces this distinction).
  • Is the headline verdict consistent with the per-entry codes (any SERIOUS → FAIL; any unverified without SERIOUS → UNVERIFIED)?
  • Did every unverified entry get the WebSearch fallback and a final verdict?

References

  • alterlab-research-pipeline/agents/integrity_verification_agent.md — canonical Five-Type Taxonomy, compound-deception patterns, and the Lin et al. (2020) mashup case study this skill is built to catch.
  • alterlab-research-pipeline/references/claim_verification_protocol.md — Phase E full-text claim-vs-source verification (beyond the abstract-level SH triage here).
  • shared/schemas/integrity_report.schema.json — the integrity-report shape whose citation_integrity_score / fabrication_risk_score this skill mirrors.
  • examples/caught_hallucination_walkthrough.md — end-to-end run on a real mashup.
  • Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13, 14045. https://doi.org/10.1038/s41598-023-41032-5
  • Ansari, S. (2026). Compound deception in elite peer review: A failure mode taxonomy of 100 fabricated citations at NeurIPS 2025 (arXiv:2602.05930). https://arxiv.org/abs/2602.05930
  • Rittman, M. (2025, January 29). Retraction Watch retractions now in the Crossref API [Blog post]. Crossref. https://www.crossref.org/blog/retraction-watch-retractions-now-in-the-crossref-api/

Part of the AlterLab Academic Skills suite.