SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslight-literature-search

light-literature-search

Writing & research

View repository

Updated 10-16 06:23

About this skill

Material Search and Standardized Organization

Goal

Help users truly and quickly understand a direction, rather than piling up search results. Produce structured materials that can be directly used for writing reviews, proposing ideas, and making plans.

Input Clarification (if missing, ask first or make reasonable assumptions)

Research direction, core problem, keywords (Chinese and English), target journal/conference, project background, time range, language preference, depth (quick overview/medium/exhaustive).

Search Strategy

  1. Multi-source in parallel: English papers (arXiv/OpenAlex/Semantic Scholar/Crossref), biomedical papers (PubMed/Europe PMC, see below "Prioritize free academic APIs without keys" and biomedical discipline), Chinese papers (prioritize OpenAlex/Crossref to search Chinese journals by ISSN; CNKI/Wanfang/VIP/CSCD have no free APIs, use institutional access or browser to retrieve metadata, see below "Chinese Literature Search"), patents (Google Patents/CNIPA/Espacenet/Lens), standards, policies, datasets (HF/Kaggle/OpenML), open source (GitHub/PwC), competition solutions, industry reports, technical blogs, official documentation.
  2. Multi-angle sweep(multi-modal sweep): search separately by concept, by method, by author/team, by time, and by citation relationships, to complement blind spots.
  3. Query construction: provide the actually usable query for each database (Boolean logic, field restrictions, synonym expansion, Chinese-English correspondence).
  4. Snowballing: perform forward (cited by) + backward (references) tracking from representative works.

Prioritize free academic APIs (most require no key; OpenAlex requires a free key starting in 2026, see references "OpenAlex access source of truth" for access specifications. Endpoints are detailed in references.md; verify again before implementation)

  • OpenAlex https://api.openalex.org/works：search= full-text search;filter= combination (from_publication_date:、cited_by_count:>N、is_oa:true、type:article、primary_topic.id:、authorships.author.id:)；sort=cited_by_count:desc；select= omit fields; for deep paging use cursor=*(page upper limit 10,000); include mailto= to enter the polite pool.group_by= can do one-click annual/journal distribution. Abstracts are inverted indexes and need to be restored.Access requires a free key; for quota/rate-limit specifications see the references "OpenAlex access source of truth" section.
  • Crossref https://api.crossref.org/works：query.bibliographic/author/title=；filter=from-pub-date:,type:journal-article；sort=is-referenced-by-count&order=desc; deep paging cursor=*; include mailto= to speed up. Serves as the "source of truth" for DOI normalization and deduplication.
  • Semantic Scholar https://api.semanticscholar.org/graph/v1：/paper/search?query=&fields=title,abstract,year,authors,citationCount,influentialCitationCount,externalIds,openAccessPdf,tldr; for exhaustive database sweeping use /paper/search/bulk(returns token to continue paging); for snowballing use /paper/{id}/citations and /references(see isInfluential); batch details /paper/batch(POST). Anonymous rate limits are strict; for large volumes apply for x-api-key。
  • arXiv http://export.arxiv.org/api/query：search_query=ti:/au:/abs:/cat:/all: + Boolean;start/max_results(≤100) paging;sortBy=submittedDate&sortOrder=descending; request interval ≥3 seconds.
  • PubMed E-utilities https://eutils.ncbi.nlm.nih.gov/entrez/eutils/: two-step esearch.fcgi?db=pubmed&term=... to get PMID → efetch.fcgi/esummary.fcgi to get details;term supports [MeSH Terms]/[tiab]/[ti]/[au]/[dp]；retmax/retstart paging, for large sets use usehistory=y+WebEnv/query_key; recommended to include &email=&api_key=(without key 3 req/s, with key 10 req/s).MeSH controlled vocabulary search and Clinical Queries clinical filters are unique to this source and cannot be replaced by OpenAlex.
  • Europe PMC https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=&format=json&resultType=core: completely key-free;pageSize(≤1000)+cursorMark(pass *, respond nextCursorMark continued translation); returns directly containing abstractText、pmid/pmcid/doi、isOpenAccess/inEPMC/hasPDF、citedByCount；/{source}/{id}/citations and /references for snowballing; covers MED+PMC+PPR.
  • Cross-source deduplication anchor: prioritize merging by DOI; if no DOI, then by arXiv id / normalized title+authors+year. Citation counts cannot be directly compared across databases (Crossref underestimates, S2 includes estimates, OpenAlex uses a different criterion, Europe PMC/PubMed yet another criterion), so note the source.
  • Biomedical discipline (mandatory): for biomedical/clinical/systematic review directions,PubMed+MeSH search strings are mandatory(aligned with the already-claimed PRISMA); you cannot rely solely on OpenAlex/S2 full-text search to pass as a standard search; MeSH controlled vocabulary and Clinical Queries clinical filters are capabilities unique to PubMed. Europe PMC serves as a key-free supplement (directly provides abstract + open access flag + citation endpoints).Preprints at the forefrontsupplement bioRxiv/medRxiv API (api.biorxiv.org, key-free,/details/ to retrieve metadata,/pubs/ to check whether it has been formally published; preprints must be labeled with a credibility tier, see references "bioRxiv/medRxiv API" for details). Search four sources (OpenAlex+Europe PMC+PubMed+bioRxiv) together to cover complementary blind spots.

Chinese-language literature search (mandatory, do not only do English)

A large amount of Chinese core output is in databases such as CNKI/Wanfang, but these databaseshave no public free API; however OpenAlex/Crossrefalready index a large number of Chinese journals(directly matchable by ISSN), and should serve as the low-barrier mainstay (OpenAlex requires a free key, Crossref is key-free), with institutionally accessed Chinese databases as a supplement.

  • OpenAlex for Chinese journals (already tested and usable): locate the source by journal name/ISSN then pull works. Example: Chinese Journal of Computers = https://api.openalex.org/sources/issn:0254-4164(tested OpenAlex ID S4210175330、country_code:CN, 1264 papers), then GET /works?filter=primary_location.source.id:S4210175330&sort=cited_by_count:desc。pitfall: OpenAlex often stores the title/abstract of Chinese papers as English translations,group_by=language may show en instead of zh, so searching by source.id is more reliable than by language:zh .
  • Crossref for Chinese journals (already tested and usable)：https://api.crossref.org/journals/0254-4164(tested hits for "Chinese Journal of Computers", publisher China Science Publishing & Media, 1264 DOI);/journals/{ISSN}/works?query= search within the journal. First use a list of Chinese core journal ISSNs to probe sources in bulk.
  • CNKI / Wanfang / VIP / CSCD (no public API, web page or institutional subscription only): all require institutional/IP authorization or manual web page searching, with no external free API. Use browser-use / agent-browser to do "human-like browsing" to obtainmetadata/abstracts/links(comply with robots/ToS, do not scrape full text); or have the user export records under an institutional account (EndNote/RefWorks/NoteExpress/RIS format) and then import them. CSCD (Chinese Science Citation Database) citation data is likewise only available on the institutional side; mark it as "not available from free sources".
  • Baidu Scholar / Google Scholar (no official API): web page only, strong anti-scraping; use only as a discovery entry point; after a hit, go back to OpenAlex/Crossref to verify metadata by DOI or journal name before entering it into the table, and do not directly trust the citation counts on their pages.
  • Suggestions for Chinese-language subject term search: ① Try both Chinese and English synonyms (e.g., "大语言模型/大模型/LLM/large language model"); ② Expand using the Chinese Thesaurus/subject standard terms + high-frequency keywords from core journals in the same field; ③ Cover both simplified and traditional Chinese, full-width and half-width characters, and proper noun abbreviations; ④ Prioritize locking onto the scope of Chinese core journals (Peking University Core/CSCD/CSSCI); you may first build an ISSN list of target journals and then search journal by journal.
  • GB/T 7714 citation format: Chinese entries are described according to GB/T 7714-2015. Journal example:主要责任者. 题名[J]. 刊名, 年, 卷(期): 起止页码.; if there is a DOI, add at the end DOI: 10.xxxx/xxxx.. Document type identifiers: monograph [M], journal [J], conference proceedings [C], dissertation [D], report [R], standard [S], patent [P], electronic resource [EB/OL]. When entering into the table, Chinese and English entries should be described according to their respective standards (Chinese GB/T 7714, English APA/IEEE, etc.) and noted.

Use a browser agent to retrieve data from sources without an API

When CNKI/Wanfang web pages, government standards/policy sites, Google Scholar, etc. have no open API or have anti-scraping measures, use browser-use(local Python+Playwright) or Browserbase/Stagehand agent-browser(cloud,observe→act→extract extract structured fields) to retrieve metadata through "human-like browsing". Retrieve only metadata/abstracts/links, comply with robots/ToS and copyright; results are non-deterministic and must be verified twice.

Supplementary web search

For official documents, technical blogs, project pages, and industry reports not covered by arXiv/academic repositories: use Exa(/search neural/keyword + contents to retrieve highlighted body text,/findSimilar use web pages as seeds for snowballing) or Parallel Search(optimized for LLMs, returns ranked excerpts). Web results must always be verified back against DOI/OpenAlex metadata before being entered into the table.

Organization workflow (cannot be omitted)

  • Screening: relevance threshold, remove obviously off-topic items. For review-type tasks, keep a trail according to the PRISMA idea: record each database's search query and hit count → deduplicated count → how many removed by title/abstract screening → how many removed by full-text screening → final included, fully reproducible throughout.
  • Deduplication: across databases, prioritize by DOI; if no DOI, merge by arXiv id / normalized title+authors+year.
  • Classification: group by subdirection/method family/task/time.
  • Credibility assessment: source tier (top journal/top conference > ordinary journal > preprint > blog), citation count (note the source database), team, whether peer-reviewed, whether reproducible. S2's influentialCitationCount/tldr can assist with quick screening, but it is a model estimate and is not hard evidence.
  • Importance ranking: prioritize foundational/milestone/SOTA/review/most relevant. You can use OpenAlex group_by to produce annual/journal distributions to assist in judging热度 and context.

Runnable scripts (in this directory scripts/, all actually run with real python, APIs actually curl with HTTP codes recorded)

When there is no network, all scripts fall back to synthetic samples and print [OFFLINE], ensuring the pipeline is verifiable.

  • scripts/search_normalize.py: urllib directly connects to OpenAlex+Crossref, restores abstract_inverted_index, deduplicates and merges across sources by DOI, sorts by citations, and outputs the bibliography table JSON+Markdown. python scripts/search_normalize.py "dairy goat behavior" --per-page 10(measured OpenAlex=200 Crossref=200). Regular tracking: add --from-date YYYY-MM-DD(only take those published after that date) + --known-dois 已读库.txt(deduplicate by DOI to output new additions) for incremental reruns; for the protocol, see references "文献定期追踪协议".
  • scripts/verify_citations.py: DOI content negotiation goes back to Crossref/doi.org to verify metadata, compare title similarity/year/first author, mark VERIFIED/METADATA_MISMATCH/DOI_NOT_FOUND(疑似幻觉)/NO_DOI。 python scripts/verify_citations.py 10.1038/s41597-023-02555-8(measured real DOI=200, fake DOI=404).
  • scripts/cn_journal_probe.py: read assets/cn_core_issn.csv Batch-probe OpenAlex source volume (id/works_count/cited_by). All 11 seed ISSNs tested returned HTTP 200.
  • scripts/snowball.py: Citation snowballing. Do one/two-hop neighbor crawling from seeds (DOI/OpenAlex workid/title)—backward references (OpenAlex referenced_works batch backfill or S2 /paper/{id}/references), forward citations (OpenAlex filter=cites:{workid}&sort=cited_by_count:desc or S2 /paper/{id}/citations); deduplicate (reuse _norm_doi/_norm_title), sort by citation count descending, tag edge type (reference/citation) and isInfluential, export JSON+Markdown.python scripts/snowball.py 10.1016/j.compag.2021.100001(use [OFFLINE] when no network).
  • scripts/prisma_flow.py: Systematic review PRISMA 2020 process traceability and count verification (for systematic review/meta-analysis). Input the counts JSON for each stage (hits per database/deduplication/title-abstract screening/full-text exclusion by reason/final inclusion), the scriptcross-checks and verifieswhether the numbers at each stage are self-consistent (previous stage − exclusions = next stage, catching arithmetic errors of "vanishing out of thin air/extra appearing", which reviewers always check), and produces the structured data needed for the PRISMA flow diagram + a text version of the process for m09 to plot and m07 to write. It only verifies count consistency, not making screening judgments for you.python scripts/prisma_flow.py --selftest(including counterexample detection) / --counts counts.json --out prisma.json。
  • assets/cn_core_issn.csv: Real ISSN seeds of Peking University Core/CSCD core journals (each verified one by one via curl against OpenAlex).
  • assets/litreview_template.md + assets/method_card.md: Review + method card fill-in-the-blank skeleton (6-piece set).
  • examples/worked_example_dairy_goat.md: End-to-end PICO→query→hits→deduplication→screening→verification traceability example.

Quantitative grading of paper importance (age × citations)

For citation counts, first note the source database (OpenAlex/Crossref/S2 have different criteria and cannot be compared directly); useannual average citations = citations / max(1, current year - publication year) to smooth out age, then combine with absolute volume and role to grade. The table below gives empirical thresholds (large field differences: CS/biomedicine tends high, niche animal husbandry/humanities tend low; calibrate by subfield):

Role Age Typical citations/annual average citation characteristics Judgment signal
Foundational ≥8 years Extremely high cumulative citations (field top 1%), annual average still ≥50 Widely cited by subsequent methods as a starting point, pioneered the problem/paradigm
Milestone 3–8 years Annual average citations ≥30, entered subfield top Proposed a widely reused method family/dataset/benchmark
SOTA / Rising star ≤3 years Absolute citations not necessarily high, but annual average growth fast, topping leaderboards Ranks high on leaderboard, densely cited by the latest work
Survey Any High citations but second-hand Suitable for quickly building context, not hard evidence of originality
Most-relevant Any Citations may be low Best fit with this topic's problem/data/method, prioritize close reading
Long tail/questionable Any Annual average citations <1 and not a new work Adopt with caution, check whether peer-reviewed/reproducible

Use OpenAlex group_by=publication_year to produce a yearly distribution to help see popularity inflection points; low citations for new work ≠ unimportant (look at growth rate and venue tier).

Standard output

  1. Literature table (fields: title, authors, year, venue, level, core contribution, method, dataset, citations, link, credibility, relevance).
  2. Research lineage (timeline + evolution logic: how the problem was raised → how it was solved → what remains).
  3. Representative method cards (see db03 method_card fields).
  4. Pros and cons comparison table.
  5. Reusable resource list (open-source libraries, datasets, templates, leaderboards, including license).
  6. Gaps and opportunities (current gap, fed directly to m03).

Standard artifacts: the above synthesis is persisted as docs/literature_review.md(handoff artifacts for m03/m04/m07/m10, naming see CONVENTIONS §6.1).

Compliance

Copyrighted full texts are not downloaded; only metadata/links/abstracts/notes/citation relationships are stored (see CONVENTIONS §5). All entries are given verifiable sources; DOIs are not fabricated. If search queries and endpoint parameters have not been verified on the spot, mark them as "needs verification" before use. Browser data retrieval complies with the target site's robots/ToS.

Connection

Results are written to db01/db03/db04 and the project repository db09; gaps are handed to m03 idea-generation. After the literature table is ingested, it is recommended to hand it to light-citation for citation verification to avoid hallucinated citations. For long-term projects, you can create in the db09 project directory literature/(saving search queries + DOIs of the read library), and periodically re-run incrementally according to the "literature periodic tracking protocol" (see references), continuously keeping an eye on new literature in the direction.

For the real endpoints, parameters, rate limits, and known pitfalls of specific tools/APIs, see the same directory references.md。