external-gitcode-ascend-arxiv-recommendation-npu

⚠️ Note: This is the public version, comes with environment variable configuration, and supports custom paths:

• ARXIV_RECO_DIR: Paper storage directory (default: ./arxiv-recommendation-models）
• NPU_MIGRATION_SKILL_PATH: npu-model-migration skill path (default: ../npu-model-migration）

arxiv-recommendation-npu automated pipeline

keywords: - arxiv - recommendation system - NPU - papers - migration - collaborative filtering - recommendation

arxiv-recommendation-npu automated pipeline

Automated recommendation system paper discovery and task generation.

Trigger conditions

When the user requests:

• "arxiv recommendation papers"
• "Help me query the latest recommendation papers and adapt them to NPU"
• "latest recommendation papers npu"
• "Find recommendation papers within one week"
• "Find recommendation-related papers from the past month"

Time range rules：

• User explicitly specifies a time range → search by the user-specified number of days
  • Example: "Find recommendation papers within one week" → --days 7
  • Example: "Recommendation-related papers from the past month" → --days 30
• User does not explicitly specify → default search for the most recent 3 days
  • Example: "Find the latest recommendation papers" → --days 3

---

Prerequisites

1. Install dependencies

# 安装 deepxiv (论文搜索)
pip install deepxiv-sdk

# 验证 NPU 环境
python -c "import torch_npu; print('NPU OK')"

2. Environment check

bash scripts/check_npu_env.sh

---

Quick start

One-click execution of the complete pipeline

# 默认搜索最近3天的论文
python scripts/main.py

# 指定搜索时间范围
python scripts/main.py --days 7      # 最近7天
python scripts/main.py --days 30     # 最近30天

# 同时指定时间和结果数量
python scripts/main.py --days 7 --max-results 100

Or use the script:

# 默认3天
bash scripts/run_full_pipeline.sh

# 指定7天
bash scripts/run_full_pipeline.sh 7

---

Workflow

This skill is only responsible for discovering papers + cloning source code, NPU adaptation is completed by npu-model-migration skill .

┌─────────────────────────────────────────────────────────┐
│  arxiv-recommendation-npu skill                        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │
│  │ Step 1      │→ │ Step 2      │→ │ Step 3      │    │
│  │ 抓取论文     │  │ 检测源码     │  │ 克隆源码     │    │
│  └─────────────┘  └─────────────┘  └─────────────┘    │
│         │                                    │          │
│         │         生成迁移任务清单             │          │
│         │         (migration_task.json)       │          │
│         └──────────────────────────────────────┘          │
└─────────────────────────────────────────────────────────┘
                          ↓ 交接
┌─────────────────────────────────────────────────────────┐
│  npu-model-migration skill                              │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │
│  │ 阶段 1      │→ │ 阶段 1.5    │→ │ 阶段 4      │    │
│  │ 目标分析    │  │ 快速尝试    │  │ NPU 验证    │    │
│  └─────────────┘  └─────────────┘  └─────────────┘    │
└─────────────────────────────────────────────────────────┘

Step 1: Crawl papers

Use deepxiv to search for recommendation system-related papers.

Time range：

• When the user does not specify,default search for the most recent 3 days
• When the user specifies a time range, search by the specified number of days

# 默认搜索最近3天
python scripts/main.py

# 指定搜索最近7天
python scripts/main.py --days 7

# 指定搜索最近30天
python scripts/main.py --days 30

Or call it in the script:

python -c "
import sys
sys.path.insert(0, 'scripts')
from fetcher import search_papers, save_paper_list
from config import TODAY_DIR

# 默认3天
papers = search_papers(days=3, max_results=50)
# 或指定天数
papers = search_papers(days=7, max_results=50)

save_paper_list(papers, f'{TODAY_DIR}/paper_list.md')
print(f'找到 {len(papers)} 篇论文')
"

Search keywords：recommendation, recommender system, collaborative filtering, CTR prediction

Output：./arxiv-recommendation-models/{date}/paper_list.md

Note: paper_list.md is an intermediate result record, convenient for review. Actual cloning uses the in-memory paper data (including enrich_papers_with_details supplemented via github_url field).

---

Step 2: Detect source code

Extract GitHub links from the papers and verify whether the repository has executable code.

python -c "
import sys
sys.path.insert(0, 'scripts')
from fetcher import search_papers
from source_detector import filter_papers_with_code

papers = search_papers(days=7, max_results=50)
papers_with_code = filter_papers_with_code(papers)
print(f'其中 {len(papers_with_code)} 篇有可用源码')
"

Verification rules：

• Prioritize checking the paper's github_url field (extracted during the enrich stage)
• Fallback: extract GitHub links from the paper's comments and abstract extract GitHub links
• Call GitHub API RecursivelyCheck that the repository has at least 3 .py files(including subdirectories)
• Exclude empty repositories with only README

---

Step 3: Clone source code & create migration tasks

For each paper with source code:

1. Clone the repository locally
2. Create migration task list (migration_task.json)
3. Wait for npu-model-migration-skill to process

python -c "
import sys
sys.path.insert(0, 'scripts')
from source_detector import clone_repo

github_url = 'https://github.com/owner/repo'
model_dir = './arxiv-recommendation-models/2026-04-13/models/repo'

result = clone_repo(github_url, model_dir)
print(result)
"

Migration task list (migration_task.json): automatically generated in the model directory, detailed format see references/output-structure.md

---

Step 4: Generate report

python -c "
import sys
sys.path.insert(0, 'scripts')
from reporter import generate_daily_report

generate_daily_report(papers_with_code, results, 'daily_report.md')
"

Output：./arxiv-recommendation-models/{date}/daily_report.md

---

Connect to the npu-model-migration skill

Next-step guidance in the report

daily_report.md will contain information about the papers to be migrated and next-step operation guidance; for the detailed format, see references/output-structure.md

Manually invoke npu-model-migration

1. View the report and find the model directory to be migrated
2. Enter the model directory
3. Refer to the npu-model-migration skill for migration:
   • Stage 1.5: Quick attempt (transfer_to_npu)
   • Stage 4: NPU verification

---

Configuration description

Modify scripts/config.py：

Configuration item	Description	Default value
BASE_DIR	Paper storage root directory	./arxiv-recommendation-models
ARXIV_KEYWORDS	Search keywords	Recommendation system related
MIN_PY_FILES	Minimum number of .py files	3
MAX_RETRIES	Maximum retry count	3
GITHUB_MIRROR	GitHub mirror	ghproxy (domestic acceleration)

---

Output structure

./arxiv-recommendation-models/{YYYY-MM-DD}/
├── main.log              # 运行日志
├── paper_list.md         # 论文列表 (Step 1-2)
├── daily_report.md       # 适配报告 (包含迁移指引)
└── models/               # 模型源码
    └── {model_name}/
        ├── migration_task.json  # 迁移任务清单
        └── [原始代码...]

---

File structure

arxiv-recommendation-npu/
├── SKILL.md                      # 本文件
├── references/                   # 参考文档
│   ├── paper-filtering.md        # 论文筛选规则
│   ├── npu-adaptation.md         # NPU 适配指南 (指向 npu-model-migration skill)
│   └── output-structure.md       # 输出结构说明
└── scripts/                      # 可执行脚本
    ├── main.py                   # 完整流水线入口
    ├── config.py                 # 配置文件
    ├── fetcher.py                # 论文抓取
    ├── source_detector.py        # 源码检测与克隆 + 创建迁移任务
    ├── reporter.py               # 报告生成 (包含迁移指引)
    ├── utils/                    # 工具模块
    │   ├── logger.py
    │   └── date_utils.py
    ├── run_full_pipeline.sh      # 完整流水线脚本
    └── check_npu_env.sh          # 环境检查脚本

---

Dependent Skill

Skill	Responsibility	Location
arxiv-recommendation-npu	Discover papers, clone source code, create tasks	This skill
npu-model-migration skill	NPU adaptation verification	../npu-model-migration/

Connection method: arxiv-recommendation-npu generates migration_task.json, and the user invokes the npu-model-migration skill according to the report guidance.

---

Common issues

Issue	Solution
Clone failed	Check the network, or manually git clone and then place it in the models directory
Do not know how to use the npu-model-migration skill	View ../npu-model-migration/SKILL.md
Want to process only specific papers	Modify scripts/main.py in papers_with_code list to manually specify

---

Reference documents

• references/paper-filtering.md - Detailed explanation of paper filtering rules
• references/npu-adaptation.md - NPU adaptation guide (points to the npu-model-migration skill)
• references/output-structure.md - Output structure description
• npu-model-migration-skill: ../npu-model-migration/SKILL.md