SuperAPI AI Directory

ingest

General & other

View repository

Updated 10-15 23:44

About this skill

Ingest skill

Core objective

You are maintaining a LLM Wiki(Obsidian knowledge base).raw/ directory is the "pending inbox",wiki/ is the "compilation output layer". This skill is responsible for compiling raw material into the knowledge base and archiving it automatically.

Directory structure

  • raw/ — user-defined category structure (can be adjusted dynamically)
  • raw/09-archive/ — Archive directory for processed files, reading is prohibited(fixed, hardcoded)
  • wiki/concepts/ — concepts (frameworks, methodologies, theories)
  • wiki/entities/ — entities (people, companies, tools, products)
  • wiki/sources/ — material summaries
  • wiki/syntheses/ — comprehensive research reports

Trigger conditions

  1. User executes /ingest — scan raw/ all subdirectories (excluding 09-archive/), find pending files
  2. User executes /ingest <path> — process only the specified file
  3. Implicit trigger — when the user says "ingest this material into the knowledge base" or "import this article", execute automatically

Workflow

Pre-check: scan pending files

Key guidelines:

  1. Use Glob to scan raw/**/*.{md,pdf,txt}(automatically excluding raw/09-archive/)
  2. Read raw/.processed.json, identify processed files
  3. Create a processing checklist for each pending file, ensure no file is missed
  4. Process in alphabetical order by filename, to make progress easier to track

Processing checklist example:

📋 待处理文件清单:
- [ ] raw/01-articles/文件A.md (ID: file-a)
- [ ] raw/01-articles/文件B.md (ID: file-b)
- [ ] raw/02-papers/文件C.pdf (ID: file-c)

Important: Before starting processing, show this checklist to the user and have the user confirm the list of files to be processed.


For each pending source file, strictly follow the steps below:

Step 1: Read the source file

  • If it is a .md file: use the Read tool to read the content in full
  • If it is a .pdf file: use the Read tool to try to extract the text. If it cannot be extracted or the content is empty, instead record the file metadata (filename, page count)
  • Checkpoint ✓: Confirm that the file content has been fully read, and record the file path and word count

Step 2: Content analysis and ID generation

Generate a stable ID:

  • Extract from the filename (remove extension and path)
  • Convert to kebab-case
  • Example:raw/01-articles/Attention Is All You Need.md → ID: attention-is-all-you-need
  • Checkpoint ✓: Confirm that the ID has been generated and does not already exist in .processed.json (to avoid duplicate processing)

Extract from the source file:

  • Core gist — what this material is about (1-2 sentences)
  • Entities — specific nouns such as people, companies, tools, products, etc. (list all entities)
  • Concepts — abstract nouns such as frameworks, methodologies, theories, etc. (list all concepts)
  • Key tags — 3-5 fine-grained tags (prefer reusing existing tags from the index.md tag cloud)

If the content is not in Chinese, translate it into Chinese.

Checkpoint ✓: Confirm that the entity list, concept list, and tag list have been extracted

Step 3: Knowledge networking (dynamic Top-K)

Read the master index:

Read wiki/index.md

Intelligently locate candidate pages:

  1. Use the entities, concepts, and tags extracted in Step 2 as keywords
  2. Match related pages in index.md (no limit on the number)
  3. Sort by relevance and dynamically determine the number to update:
    • Simple blog/short article(< 1000 words): 2-5 pages
    • Technical article/paper(1000-5000 words): 5-15 pages
    • Book chapter/long article(> 5000 words): 15-30 pages
  4. If more than 20 pages need to be updated → ask the user whether to process them in batches

Read only the content of the Top-K pages(avoid a full scan)

For each entity and concept extracted in Step 2:

Target directory:

  • Entity → wiki/entities/
  • Concept → wiki/concepts/

Processing logic:

  1. Page does not exist → create a new page according to the Frontmatter specification in CLAUDE.md
  2. Page already exists → read the existing content and incrementally merge in the new information
  3. Conflict found → handle it according to the conflict type:
    • Temporal conflict(old version vs new version) → update directly, and annotate in the page ## 历史版本
    • Opinion conflict(statement A vs statement B) → create a new ## 知识冲突 block in the page
    • Complex conflict(cannot be determined) → in wiki/_conflicts/ create a conflict file, pause, and ask the user

Page template:

---
type: entity | concept
tags: [标签1, 标签2, 标签3]
source_id: 稳定ID
created: 2026-04-29
updated: 2026-04-29
---

## 定义
[对该实体/概念的定义]

## 关键信息
[从源文件中提取的详细信息]

## 关联连接
- [[摘要-source-slug]] — 来源
- [[RelatedEntity]] — 相关实体

Tag usage principles:

  • Prefer to reuse existing tags
  • 3-5 tags per page
  • Use Chinese or English for tags, and keep them consistent

Checkpoint ✓: Confirm that all entity and concept pages have been created or updated, and record the list of created/updated pages

Step 4: Create a source summary

In wiki/sources/ create a Markdown file:

---
type: source
tags: [标签1, 标签2, 标签3]
source_id: 稳定ID
created: 2026-04-29
updated: 2026-04-29
---

## 核心摘要
[3-5句话的核心总结]

## 关联连接
- [[EntityName]] — 关联实体
- [[ConceptName]] — 关联概念

Use kebab-case for the file name:摘要-{稳定ID}.md

Checkpoint ✓: Confirm that the source summary page has been created

Step 5: Update the index and log

Update the master index wiki/index.md

Group by type and add the new pages under the corresponding category:

  • Format:- [[页面名称]] — 一句话描述
  • Update the tag cloud: recount the usage frequency of all tags

Checkpoint ✓: Confirm that index.md has been updated and the new pages have been added to the index

Update the operation log wiki/log.md(Append-only):

## [YYYY-MM-DD] ingest | 操作简述
- 变更: 新增 [[PageName]] (N个); 更新 [[ExistingPage]] (M个)
- 标签: #标签1 #标签2 #标签3
- 冲突: 无 (或: 时间性冲突已更新; 或: 复杂冲突已生成报告到 _conflicts/)
- 源文件: raw/xx/文件名.md

Checkpoint ✓: Confirm that a new record has been appended to log.md

Step 6: Update the processing status and archive the source file

Update raw/.processed.json:

Use a stable ID (extracted from the file name) and record the processing status:

{
  "文件ID": {
    "original_path": "raw/01-articles/xxx.md",
    "archived_path": "raw/09-archive/xxx.md",
    "processed_at": "2026-04-29T14:30:00Z",
    "pages_created": ["页面1", "页面2"],
    "pages_updated": ["页面3"]
  }
}

Timestamp format: Use ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ)

Checkpoint ✓: Confirm .processed.json has been updated

Archive the source file:

Key criterion: The source file can be archived only when all of the following conditions are met:

  • ✓ The sources page has been created
  • ✓ Entity/concept page created or updated
  • ✓ index.md updated
  • ✓ log.md updated
  • ✓ .processed.json updated

Use the Bash tool to perform the move operation:

# 确保归档目录存在
mkdir -p raw/09-archive

# 移动文件(使用相对路径)
mv "raw/01-articles/文件名.md" raw/09-archive/

Verify archiving:

# 验证文件已移动
test -f "raw/09-archive/文件名.md" && echo "归档成功" || echo "归档失败"

Checkpoint ✓: Confirm that the source file has been successfully moved to raw/09-archive/, and that the file no longer exists in its original location

Modifying the text inside the source file is absolutely prohibited.

Step 7: Completion Prompt

For each file that has been processed, output:

✅ 已完成文件处理:raw/xx/文件名.md
- 稳定ID: xxx
- 新增页面:N 个
- 更新页面:M 个
- 已归档到:raw/09-archive/

After all files have been processed, output a summary:

🎉 所有文件摄入完成!

📊 总计:
- 处理文件:X 个
- 新增页面:N 个
- 更新页面:M 个
- 源文件已归档到:raw/09-archive/

📝 下一步:使用 `/sync` 提交到 Git

Conflict Handling

When conflicts between new and old knowledge are found, handle them automatically according to the conflict type:

1. Temporal conflict (old version vs new version)

Characteristics: Descriptions of the same fact at different points in time, where the new information is clearly more up to date Handling: Directly update to the new version, and add a ## 历史版本 block in the page to record the old information

2. Opinion conflict (Statement A vs Statement B)

Characteristics: Different sources hold different views on the same issue, and it is impossible to determine which is correct Handling: Create a new ## 知识冲突 block in the page, retaining both statements and comparing them

3. Complex conflict (type cannot be determined)

Characteristics: Contradictions involving multiple pages and multiple dimensions, which the LLM cannot decide on automatically Handling:

  1. In wiki/_conflicts/ create a conflict file:conflict_YYYY-MM-DD_主题.md
  2. The file content includes:
    • Conflict description
    • Pages and sources involved
    • Suggested solution
    • Decision options (for the user to check)
  3. Pause the ingest process and report the conflict location to the user
  4. Wait for the user's decision before continuing

Detailed Guidelines

File Processing Guidelines

  1. Batch processing: When processing multiple files, process them one by one in alphabetical order, and archive each file immediately after it is completed
  2. Error recovery: If processing a file fails, record the error information and continue processing the next file
  3. Progress tracking: When processing multiple files, update the processing checklist in real time and mark completed files
  4. Filename encoding: When processing Chinese filenames, use relative paths and the cd command to avoid encoding issues

Page Creation Guidelines

  1. Naming conventions:
    • Entities: Use the original name (Chinese or English), maintaining TitleCase
    • Concepts: Use Chinese names, maintaining TitleCase
    • Sources: Use the 摘要-{kebab-case-id} format
  2. Duplicate check: Before creating a page, check whether a page with the same name or a similar page already exists
  3. Link integrity: Each page must include a ## 关联连接 section, linking at least to the source page
  4. Tag consistency: Reuse existing tags first, and avoid creating new tags with the same meaning

Index Update Guidelines

  1. Accurate categorization: Ensure pages are added to the correct category (Concepts/Entities/Sources/Syntheses)
  2. Concise descriptions: Keep each entry's description within 10-15 characters
  3. Tag statistics: The tag cloud is sorted in descending order of usage frequency, in the format #标签 (数量)
  4. Accurate counts: Each category's count must match the actual number of pages

Logging guidelines

  1. Time format: Use the [YYYY-MM-DD] format
  2. Operation type: Clearly indicate the operation type (ingest/query/lint/sync)
  3. Change details: List all newly added and updated pages, using wikilink format
  4. Source file traceability: Record the original path of the source file for traceability

Archive verification guidelines

  1. Pre-move check: Confirm that all related pages and indexes have been updated
  2. Post-move verification: Use the ls or find command to verify that the file has been moved successfully
  3. Original location cleanup: Confirm that the file no longer exists in the original location
  4. Archive record: Record the archive path and time in .processed.json record the archive path and time in

Notes

  • Never read any files under raw/09-archive/ any file under
  • All wiki pages must include the ## 关联连接 section, and no orphan pages may be created
  • Write all content in Simplified Chinese
  • Use TitleCase for entity names, and kebab-case for concepts and sources
  • Use the Glob tool to scan files, the Read tool to read content, and the Write/Edit tools to create/update pages
  • Use the Bash tool to move files to the archive directory
  • Only read the master index and related pages; do not perform a full scan
  • When processing multiple files, you must complete them one by one and must not skip any file
  • After each file is processed, archive it immediately; do not wait until all files are processed before archiving