SuperAPI AI Directory
Partner中文AI newsFavorites
Skillstop-journals-methods-scout

top-journals-methods-scout

General & other

View repository

Updated 10-16 07:21

About this skill

Top Journals Methods Scout

A top journal methods scanning tool for course planning and methods teaching. Default input:say nothing—— This skill automatically uses a list of top journals across 14 disciplinary categories + the past 24 months + a separate AI chapter.

Design Reality (read this first)

During the first trial run, the data scale was underestimated—top journal publications across 14 disciplinary categories over the past 24 months number about 10,000 papers(the high publication volume of journals such as AHR/Q&Q is the main reason). Performing per-paper LLM deep coding on 10k papers is neither economical nor necessary. Therefore, this skill'sactual main pathis:

  1. Heuristic rule coding（heuristic_classify.py)——tag the entire database with L1 family labels + AI identification, with fast coverage
  2. Python generates the statistical skeleton（generate_outputs.py)——Excel matrix + Markdown automatic statistics section
  3. The main agent writes the analysis layer(add a methods_report_analysis.md)——use heuristic statistics as the basis to write the abstract, emerging methods, AI chapter, and teaching module suggestions

LLM deep coding is used only for specific subsets (such as when the user focuses on a certain discipline, or for selecting 50 papers as teaching candidates).

Dependencies

pip install openpyxl

The OpenAlex API is free and requires no key; the script has built-in retry and rate limiting.

Step 1: Requirements confirmation (≤ 3 questions)

Confirm with the user using a concise checklist (do notuse open-ended questionnaire-style follow-up questions):

  1. Disciplinary scope: Default is all 14 categories; do they want only a few of them?
  2. Time window: Default is the past 24 months; should it be adjusted to 12/36/48 months?
  3. AI focus: Default is a separate AI chapter + conventional methods in parallel; is it a pure AI methods scan?

If the user replies "default," proceed directly to the next step.

Be candid with the user about the cost: Scanning all disciplines over 24 months covers about 10k papers; retrieval takes 5-10 minutes, heuristic coding 1 minute, statistics generation 10 seconds, and analysis writing is completed by the main agent. Total time is about 15-20 minutes.

Step 2: OpenAlex retrieval

mkdir -p /tmp/methods_scout
python3 /Users/songyiping/.claude/skills/top-journals-methods-scout/scripts/fetch_journal_papers.py \
  --journals-file /Users/songyiping/.claude/skills/top-journals-methods-scout/references/journals.json \
  --months 24 \
  --out /tmp/methods_scout/papers.json

If only certain disciplines are needed, append --disciplines sociology ai_social_intersection ...。

The script has built-in exponential backoff retry (5 times), so network fluctuations will not interrupt it. A single retrieval takes 5-10 minutes. After completion, report the total number and distribution by discipline.

Step 3: Heuristic method coding (main path)

python3 /Users/songyiping/.claude/skills/top-journals-methods-scout/scripts/heuristic_classify.py \
  --papers /tmp/methods_scout/papers.json \
  --out /tmp/methods_scout/classifications.json

Built-in rules (see details in scripts/heuristic_classify.py in METHOD_RULES and AI_OBJECT_RULES）：

  • Scan the concatenated text of title + abstract + keywords + concepts + topics
  • Match L1 labels according to rule priority (AI methods precede ordinary methods to ensure that LLM-assisted qualitative research is correctly identified)
  • AI identification: tool (uses AI) / object (studies AI) / hybrid / n/a
  • The output includes evidence snippets for rule hits (in the notes column)

Typical characteristics of coding results:

  • Unmatched rate 60-75% (humanities and history paper abstracts usually do not describe methods) — normal
  • AI recognition rate usually 5-10%
  • L1 distribution most common: qual.ethnography / qual.interview / ai.llm / quant.experiment / quant.survey

Not needed LLM codes paper by paper. If the user later requests deep coding of a subset, call the LLM again (see §Supplementary Paths).

Step 4: Generate Excel + Markdown skeleton

python3 /Users/songyiping/.claude/skills/top-journals-methods-scout/scripts/generate_outputs.py \
  --papers /tmp/methods_scout/papers.json \
  --classifications /tmp/methods_scout/classifications.json \
  --journals-file /Users/songyiping/.claude/skills/top-journals-methods-scout/references/journals.json \
  --out-dir /tmp/methods_scout/

Artifacts:

  • methods_matrix_<timestamp>.xlsx — 4-sheet matrix (methods_matrix / ai_papers / by_discipline_summary / journals_meta)
  • methods_report_<timestamp>.md — automatic statistics skeleton (including L1 frequency, discipline × method, AI subcategory distribution, journal coverage, complete appendix of AI papers)

Excel can be used independently; filtering by sheet can answer most "which paper uses which method" questions.

Step 5: Main agent writes the analysis-layer report

This step is themain agent's core contribution— what Python outputs is a skeleton; what readers want is judgment.

Important: Do not directly edit the MD generated by Python (it is a regenerable statistical artifact). Instead,create a new /tmp/methods_scout/methods_report_analysis.md, independently carrying the analytical narrative.

5.1 Sampling verification

Use a short Python snippet to quickly pull a few highly cited papers under ai_subcategory (including title/journal/DOI/citations/abstract opening); these will be the "bones" cited in the report. Example:

python3 << 'EOF'
import json
from collections import defaultdict
with open('/tmp/methods_scout/papers.json') as f:
    papers = {p['openalex_id']: p for p in json.load(f)['papers']}
with open('/tmp/methods_scout/classifications.json') as f:
    cs = {c['openalex_id']: c for c in json.load(f)['classifications']}

groups = defaultdict(list)
for oid, c in cs.items():
    if c['ai_involved'] != 'yes':
        continue
    p = papers[oid]
    groups[(c['ai_category'], c['ai_subcategory'])].append(
        (p['cited_by_count'], p['title'], p['journal'], p['doi'])
    )
for k, v in sorted(groups.items()):
    v.sort(reverse=True)
    print(f"\n{k} — {len(v)} 篇")
    for cited, t, j, d in v[:4]:
        print(f"  [{cited}] {j}: {t[:100]}")
        print(f"    {d}")
EOF

Also look at highly cited papers in methods journals (tier_map[journal]=='methods' and cited>=5) — this batch of papers is often the representative of "emerging methods".

5.2 Writing standards

Report structure (refer to /tmp/methods_scout/methods_report_analysis.md existing examples):

  1. Executive summary(300-500 words)
    • Strongest signals, three-band distribution, AI method profile, 2-3 direct recommendations for curriculum planning
  2. Emerging methods scan(3-6)
    • Each: frequency signal, representative papers (including citations + DOI), method core, relationship to existing methods, boundary conditions, pedagogical translation direction
  3. AI methods special chapter
    • 2.1 AI-as-tool overview (subcategory distribution + typical papers)
    • 2.2 AI-as-object overview (subfield analysis)
    • 2.3 Hybrid category
    • 2.4 Ethical and validity challenges of AI methods (in the form of "common reviewer concerns")
  4. Methods papers column
    • Papers in methods journals where the method itself is the contribution, presented in a table
  5. Method gaps(for the next 12-24 months)
    • Identify in top journalsnot seenbut that should be there — this is predictive analysis
  6. Pedagogical translation recommendations (core)
    • 5.1 Modules that can be launched immediately — each module includes: teaching objectives, prerequisite knowledge, core readings (2024-2026 top-journal papers + citation counts + DOI), operational task (assignment prototype, one sentence), class hours, risk notes
    • 5.2 Update points for existing courses (table: existing course | delete | add)
    • 5.3 Recommendations for cross-departmental co-construction
  7. Data limitations(must be honest)
  8. Deep-dive entry points(5 "must-read" + 5 "secondary-read")

5.3 Language red lines (specified in CLAUDE.md)

  • Ban boilerplate ("increasingly important", "helps", "enriches")
  • Every claim must be supported by a paper (give DOI or citation count)
  • Do not fabricate data; when uncertain, say so explicitly
  • Teaching suggestions must be specific to credit hours, prerequisite knowledge, and assignment prototypes
  • Chinese academic writing, formal but not rigid, reject stereotyped formulaic style

5.4 Target length

About 300-500 lines of Markdown (8,000-12,000 Chinese characters). Too short means the data was not really digested; too long means padding.

Step 6: Deliver to the user

Give the user a 5-line summary:

✓ 扫描完成：N 篇论文，M 本期刊，K 个学科
✓ Excel 方法矩阵：/tmp/methods_scout/methods_matrix_*.xlsx（4 Sheet）
✓ 统计骨架：/tmp/methods_scout/methods_report_*.md
✓ 分析报告：/tmp/methods_scout/methods_report_analysis.md  ← 先看这份
  - AI 相关 X 篇（tool: X1 / object: X2 / hybrid: X3）
  - 识别 N 个新兴方法，拟 M 个可上线教学模块

Then offer follow-up optional actions:

  • Convert to Word/PDF
  • Copy to the desktop to avoid /tmp being cleared
  • Dive deep into a key paper (use academic-work-analyzer）
  • Push skill improvements to GitHub

Supplementary path: small-batch deep coding (optional)

If the user explicitly requests deep LLM coding for a subset (for example, "write L3 operation summaries for all 50 teaching candidate papers"), the process is:

  1. Filter the target subset in Excel (for example, teaching_candidate=yes, or select manually)
  2. Read each paper's title/abstract/keywords for deep LLM coding (fill L2 fields such as data_type/sample_size/time_window_data/geographic_scope/analytic_tools/validity_strategy + L3 operation summary)
  3. Merge back into classifications.json, rerun generate_outputs.py

This step has no script automation—because it requires contextual judgment, it is suitable for the main agent to read the JSON directly and output an enhanced version of classifications.

Journal list maintenance

The default list is at references/journals.json, containing the ISSNs of about 50 top journals across 14 major discipline categories. If you need to add or remove journals, edit that JSON directly. When adding a journal, ISSNs must be included (both print and electronic; OpenAlex uses | to connect queries).

Levels already covered (tier):

  • top — journals recognized as top in their discipline
  • methods — methodology special issues
  • ai_intersect — journals at the intersection of AI and social science
  • dh — digital humanities journals
  • review — review journals

Common adjustments

  • Focus on Chinese social science: This skill has no Chinese journals; it can connect to the cnki-advanced-search skill to supplement C-journals
  • Scanning a specific topic(such as "LLM research methods"): first fetch, then in heuristic_classify.py after output, filter by keyword the abstracts of papers.json abstracts
  • Compare two periods: run fetch twice separately (with different months parameters), and when merging add period field
  • Too many Quality & Quantity papers: This journal has 862 papers, mainly applied, and its proportion of methodological innovation is lower than SMR. When screening, note that the sample quality of this journal varies—it is recommended to filter in Excel by tier=methods AND cited>=5 to obtain a high-signal subset

Design philosophy

  1. Heuristic first, LLM as fallback: a realistic choice forced by the 10k scale—rules are fast, traceable, and broad in coverage
  2. Python for statistics, agent for narrative: the skeleton can be regenerated, the narrative requires judgment, and the division of labor is clear
  3. Serious about AI classification: distinguish "using AI" from "researching AI"; their teaching implications are completely different
  4. Teaching translation is a first-class citizen: Every emerging method must address "Can this be taught? How should it be taught?"
  5. Honestly acknowledge limitations: 71% unmatched is the real situation; don't pretend to have perfect coverage