SuperAPI AI Directory
Partner中文AI newsFavorites
Skillssci-papers-downloder

sci-papers-downloder

Writing & research

View repository

Updated 10-16 09:55

About this skill

sci papers downloder

What this gives the agent

End-to-end: topic/keywords/DOI in → validated PDF files out. One pipeline, two interchangeable search backends, a layered download strategy, and concurrency.

Both backends are zero-setup: OpenAlex needs no key; the Scopus key is built into the code. The only thing the operator must provide is UNPAYWALL_EMAIL.

Agent decision policy (no-context deterministic)

Follow this top-down. Each step says which script and why.

  1. Default to OpenAlex end-to-end → skills/sci-papers-downloder/scripts/openalex_batch_download.py. Free, no key, ~250M works, returns metadata and a direct OA PDF hint. This is the right choice for ~95% of "find and download papers on X" requests.

  1. Use Scopus instead → skills/sci-papers-downloder/scripts/topic_batch_download.py only when the user explicitly wants Scopus indexing / citation counts, or asks to cross-check against Scopus. Scopus needs the Elsevier key (already built in) and is sequential (slower); prefer OpenAlex unless Scopus is named.

  2. Already have DOIs? Skip search → skills/sci-papers-downloder/scripts/download_open_access.py (--doi repeatable, or --doi-file). This is the shared downloader both end-to-end runners call.

  3. Search without downloading? → skills/sci-papers-downloder/scripts/search_openalex.py (no key) or skills/sci-papers-downloder/scripts/search_scopus.py (key built in).

Download layering (automatic, inside every download)

For each DOI the downloader tries, in order, and stops at first valid %PDF:

  1. OA direct URL (the oa_url hint from search) — highest hit rate.
  2. Unpaywall by DOI — green/gold OA across ~50k publishers.
  3. Sci-Hub via scihub-cli — on by default (--scihub-fallback auto). Recovers older closed-access papers Unpaywall can't.

Setup (minimal)

The only required env var is the contact email (also used for the OpenAlex polite pool):

export UNPAYWALL_EMAIL="<your_real_email>"      # required for OpenAlex + Unpaywall

Already wired in, no action needed:

  • Elsevier/Scopus key — built-in plaintext default in search_scopus.py (DEFAULT_ELSEVIER_API_KEY). Override anytime via --api-key or ELSEVIER_API_KEY. Rotated periodically by the operator.
  • Sci-Hub fallback — on by default. Needs scihub-cli on PATH (or it auto-bootstraps via uvx). Disable per-run with --scihub-fallback off.

Intent mapping: quantity + freshness

This is the deterministic mapping from Chinese phrasing to flags.

Quantity (default: batch)

  • "a few" / "some" / "a few is enough" → --quantity-mode few (target ~5)
  • "a batch" / "batch" → --quantity-mode batch (target ~20)
  • "as many as possible" / "the more the better" → --quantity-mode max (high caps, bounded runtime)
  • explicit number ("12 articles") → --target 12 (overrides quantity mode)
  • not mentioned → --quantity-mode batch

Freshness

  • "latest" / "recent years" / "recently" → add --latest (last 3 years + date-first sort)
  • "the last N years" → --latest --years-back N
  • "since 2023" → --from-year 2023

Combinations

  • "the latest batch" → --quantity-mode batch --latest
  • "the latest few" → --quantity-mode few --latest
  • "the latest 8 articles" → --target 8 --latest

Priority (must follow)

  1. explicit --target > quantity keywords
  2. explicit --from-year > --years-back
  3. --latest implies date-first ranking
  4. latest requested with no year → 3-year window

Recommended commands

Default: OpenAlex, no key (use this first)

python3 skills/sci-papers-downloder/scripts/openalex_batch_download.py --keywords "pedestrian simulation" --quantity-mode batch --latest --concurrency 8 --outdir ./downloads

Flags worth knowing:

  • --concurrency N — parallel downloads, default 8. Polite range 8–16; higher gets rate-limited by OpenAlex/Unpaywall.
  • --oa-only is on by default (most reliable). --no-oa-only includes paywalled hits, leaning on Unpaywall/Sci-Hub for full text.
  • --scihub-fallback {auto,off,force} — default auto. off = OA/Unpaywall only; force = skip Unpaywall, go straight to Sci-Hub.
  • --scihub-timeout S — per-DOI Sci-Hub budget (default 180). Lower it (e.g. 60) when many closed DOIs would otherwise serialize long waits.

Scopus, key built in (only when Scopus is explicitly wanted)

python3 skills/sci-papers-downloder/scripts/topic_batch_download.py --keywords "pedestrian simulation" --quantity-mode batch --latest --outdir ./downloads

Download by known DOIs

python3 skills/sci-papers-downloder/scripts/download_open_access.py --doi "10.1103/PhysRevE.51.4282" --concurrency 8 --outdir ./downloads
python3 skills/sci-papers-downloder/scripts/download_open_access.py --doi-file ./dois.txt --concurrency 8 --outdir ./downloads

Search only

python3 skills/sci-papers-downloder/scripts/search_openalex.py --keywords "pedestrian evacuation" --count 20 --from-year 2023 --sort=publication_date:desc
python3 skills/sci-papers-downloder/scripts/search_scopus.py --keywords "pedestrian evacuation" --count 20 --sort=-coverDate

Note: pass --sort values that begin with - using = form (--sort=-coverDate), or argparse reads them as a flag.

What to expect (measured behavior)

  • Gold/green OA (PLoS, Nature 10.1038, Springer 10.1007, Scientific Reports, most DOAJ): downloaded via OA-direct/Unpaywall. Typical: few+latest = ~6/8 hits in <10s.
  • Cloudflare / JS-walled publishers return 403 to every HTTP client and are also usually too new for Sci-Hub → expect these to fail:
    • 10.3390/* = MDPI
    • 10.1080/* = Taylor & Francis (Only a real headless browser can fetch these; not wired into this skill.)
  • Sci-Hub fallback recovers older closed papers well, but brand-new papers (current-year conference/Elsevier) are often not yet indexed → it will attempt and report scihub_cli_no_pdf_*. Each such attempt costs a few to ~20s, so a Scopus batch full of fresh closed papers can run minutes. If speed matters and recovery is unlikely, pass --scihub-fallback off.
  • Hard paywall, no OA, not on Sci-Hub (recent 10.1016 Elsevier, IOP): metadata only — not legally obtainable.

Fallback command resolution

download_open_access.py picks the Sci-Hub command in order:

  1. --scihub-cmd
  2. local scihub-cli on PATH
  3. uvx --from git+https://github.com/Oxidane-bot/scihub-cli.git scihub-cli

Output contract

Report:

  • query + sort + year filter
  • total hits + scanned + candidate DOI count
  • attempted + downloaded counts (+ concurrency)
  • per-DOI status / download_method (oa_direct | unpaywall | scihub_fallback) / path / error

Legal / ethics

Sci-Hub fallback is enabled by default for this operator's self-use. Copyright and access-law compliance is the operator's responsibility. To run strictly via authorized OA sources, pass --scihub-fallback off.

Resources

  • skills/sci-papers-downloder/scripts/search_openalex.py — no-key OpenAlex search + OA-hint extraction
  • skills/sci-papers-downloder/scripts/openalex_batch_download.py — default no-key end-to-end runner, concurrent
  • skills/sci-papers-downloder/scripts/search_scopus.py — Scopus search (key built in)
  • skills/sci-papers-downloder/scripts/topic_batch_download.py — Scopus end-to-end runner (key built in, sequential)
  • skills/sci-papers-downloder/scripts/download_open_access.py — shared downloader: OA-direct → Unpaywall → Sci-Hub, --concurrency