knowledge-base-management

Knowledge Base Management — Full Lifecycle Management of the Knowledge Base

Core Architecture

Three-Layer Architecture

素材库/  → Layer 1: 不可变原始素材（AI 只读）
wiki/    → Layer 2: AI 编译的结构化知识（AI 维护）
产出/    → Layer 3: 按需生成的视图（不持久化）

Core Principles: The material library is the source of truth (immutable), the Wiki is an AI-compiled projection (must be traceable), and outputs are views generated on demand (not persisted).

Vault Configuration

The vault path is configured via environment variables, not hardcoded in scripts:

export VAULT_DIR="$HOME/Documents/your-vault"   # 你的 Obsidian vault 根目录

A complete repository can uniformly configure collection, search, and source packages:

python3 tools/chubby.py init --vault "$VAULT_DIR"

The path here is the knowledge base root directory. Collection enters the root directory's 00_Inbox, and the default index is .chubby/index.sqlite. After installing this skill independently,tools/vault_index.py and tools/evidence_brief.py directly use VAULT_DIR or --vault, and do not read the repository's chubby.yaml。

Health checks can be added to a scheduler on your own; installing this skill does not create a cron.

1. Material Ingestion (Ingest)

New Material Processing Pipeline

用户丢素材 →
  ① 判断类型 → 存入素材库/对应子目录
  ② 提取关键信息 → 决定是否建 wiki 页面
  ③ 如果值得：建选题（打 SHARP 分）或更新项目页
  ④ 更新 wiki/index.md 索引

Material Grading (ABC Grading)

Grade	Criteria	Processing
A	Core methodology/framework/system	Prioritize compiling into topics
B	Insightful but not systematic enough	Alternative, serves as a supplement to A
C	Junk/duplicate/unrelated to positioning	Archive, do not proactively compile

WeChat Official Account Article Processing

Source 1: Optional external sync via wechat-article-exporter If you have already deployed this tool and a scheduler yourself, you can sync WeChat Official Account articles to 素材库/公众号文章/<公众号名>/. This skill does not include this service or scheduled tasks.

Source 2: Import articles saved on mobile Articles saved via mobile first land in a pending directory (by date), then a categorization script moves them into the material library and filters high-quality articles by domain for A+B in-depth processing.

Categorization scripts (such as sync_notes_to_kb.py) are strongly tied to each person's directory structure and are not included in this repository; just build your own according to the "Workflow" below.

Workflow：

新文章入库 →
  ① 脚本自动归类到 素材库/公众号文章/<公众号名>/
  ② URL 去重（跳过已存在的文章）
  ③ 领域关键词筛选（AI/半导体/消费/电商/品牌/投资/Agent/Skill）
     跳过纯营销、与用户领域无关的文章
  ④ 命中关键词的精选文章 → A+B 双轨处理（每日上限 5 篇）
     A层：观点提取+分类+3条Takeaway → wiki/📡 外部输入/公众号/<公众号名>/<主题>/
     B层：问题链+伴读引导 → 同上
  ⑤ 更新 wiki/index.md（外部输入 — 最新 A+B 处理 表格）

Key Principles：

Focus on high-value topics (matching user interests), do not pursue full coverage
Each wiki page must use source frontmatter to list the referenced material paths
In-depth processing limit of 5 articles/day, digest in batches
Article path format:[[../../../../素材库/公众号文章/<公众号>/<文件名>]]

2. Health Check (Audit)

Automatic Check Items

After manually running the script or configuring your own scheduler, check:

Broken links (broken wikilinks)
Missing source field
Missing frontmatter
Concepts pending consolidation
Low link density

Script：scripts/vault_health_check.py(provided by this repository, zero dependencies, pure standard library)

# 指定 vault 路径
python3 scripts/vault_health_check.py "$HOME/Documents/your-vault"

# 或读 VAULT_DIR 环境变量
export VAULT_DIR="$HOME/Documents/your-vault"
python3 scripts/vault_health_check.py --json

# 把缺 source 字段也视为问题（强溯源场景）
python3 scripts/vault_health_check.py --require-source

Check items: broken links, missing frontmatter/source/summary, orphan notes, empty directories, duplicate content, filenames containing spaces. When broken links exist, the exit code is 1, making it easy to integrate with cron / CI alerts.

Cleanup Criteria

Operation	Rule
Deduplication	Delete the copy after SHA256 or diff confirms they are identical
Put back in place	Move scattered files by type into the asset library/corresponding subdirectory
Archive	Move historical versions to `wiki/🗄️ 归档/`, do not delete
Empty directory	Delete directly. Legacy directories containing a redirect README also count as empty directories
Full repository audit	Execute Section 6 "Full Repository Audit and Batch Archiving"
Directory review	Execute Section 5 "Directory Review and Incremental Cleanup"

File Naming Conventions

✅ 选题-AI-Agent-未来方向.md    # 连字符连接，无空格
❌ 选题-DemisHassabis Agent.md  # 含空格，wikilink 断裂

Migration Notice Page Format

When document content is migrated to a new location, the old file is not deleted but replaced with a migration notice:

---
title: ⚠️ 已迁移 - 原文件名（旧版 vX.0）
type: note
tags: [已废弃, 旧版, 已迁移]
created: YYYY-MM-DD
summary: 已被 [[新位置/新文件名|新版本]] 取代。
---

Key rules：

frontmatter: ⚠️ 已迁移 heading + 已废弃 tag
Explain why the old version no longer applies (1-3 items)
Clear new link
Do not retain the old body content

3. Tool Integration (Tools)

🧠 Incremental Indexing and Unified Search

Unified entry point for the full repository:

python3 tools/chubby.py init --vault "$VAULT_DIR"
python3 tools/chubby.py ingest "替换为真实素材链接" --no-enrich
python3 tools/chubby.py search "AI Agent"
python3 tools/chubby.py search "内容策略" --mode lite

After successful collection, and before unified search, the index is synchronized automatically.init --vault points to the root directory; the old ingest --vault parameter still indicates a specific ingestion directory.

After installing this skill independently, use the bundled tools within the skill directory:

export VAULT_DIR="$HOME/Documents/your-vault"
python3 tools/vault_index.py index "$VAULT_DIR"
python3 tools/vault_index.py search "AI Agent"
python3 tools/vault_index.py semantic "内容策略"
python3 tools/vault_index.py search "品牌" --platform wechat
python3 tools/vault_index.py recent --limit 10
python3 tools/vault_index.py read "10_Sources/x/example.md" --vault "$VAULT_DIR"
python3 tools/vault_index.py stats

Standalone vault_index.py search reads the existing index; after notes change, run indexfirst. Subsequent commands need to maintain VAULT_DIR, or explicitly pass the same database:

python3 tools/vault_index.py --db /path/to/index.sqlite index "$VAULT_DIR"
python3 tools/vault_index.py --db /path/to/index.sqlite search "内容策略"

Automatically enabled when SQLite FTS5 is available; when unavailable, LIKE search is used. Default semantic-lite performs only local word and character combination ranking, without calling the model API.

Indexing rules:

Default location is $VAULT_DIR/.chubby/index.sqlite；VAULT_INDEX_DB or explicit --db can override.
index Default incremental synchronization. Vectors for unchanged notes are retained; when the actual embedding input changes, the old vector is invalidated; when a note is deleted, the corresponding index and vector are removed.
When the new default index does not exist, the old $VAULT_DIR/.chubby/vault_index.sqlite is copied and migrated, and the original file is retained. Old databases in other locations must be explicitly specified and verified as belonging to the knowledge base.
The database is bound to the knowledge base; scan or read failures roll back the synchronization, and symbolic links that go outside the knowledge base scope are rejected.
A full rebuild requires explicitly executing python3 tools/vault_index.py index "$VAULT_DIR" --rebuild, which discards existing embeddings.

📥 Import Local Materials

In the full repository,python3 tools/chubby.py import "/path/to/document.md" connects Markdown, text, or text-layer PDFs to the existing collection, reuse, and indexing workflows. --source-url can be used to specify the original web page; local attachments are copied and references rewritten, with the original files retained. PDFs require the optional pymupdf; scanned documents will not be treated as a successful import.

After installing this skill independently, it can also be used in the skill directory:

python3 tools/import_document.py "/path/to/document.md" --output "$VAULT_DIR/00_Inbox"
python3 tools/vault_index.py index "$VAULT_DIR"

The standalone import tool does not write run records for the unified CLI; MCP queries or brief still automatically synchronize the knowledge base before generation. The installation package includes this tool and the required common modules.

📎 Verbatim Source Material Package

The full repository uses a unified entry point:

python3 tools/chubby.py brief --topic "内容策略" --output reports/content-brief.md

After independent installation, run in the skill directory:

python3 tools/evidence_brief.py --vault "$VAULT_DIR" --topic "内容策略" \
  --output "$VAULT_DIR/30_Output/content-brief.md"

The material package tool first automatically synchronizes the index; when keyword retrieval returns no hits, it falls back to local semantic-lite; the generated research_brief Documents are excluded from candidates. Output Markdown and a JSON file with the same name, containing verbatim excerpts, the original source, the note's relative path, the real file line numbers, and the SHA-256.

Line numbers are counted from the first line of the source file, including frontmatter. Before exporting, check the file and excerpts again; if the source changes, they need to be regenerated. Existing outputs are rejected for overwrite by default; only an explicit --force replaces them. If there is no matching content, state that the evidence is insufficient.

The tool only organizes the original-text evidence package and the Agent task description; the topic is generated separately by the Agent. Verification of files and excerpts does not mean the facts in the original text have been confirmed. The Agent is required to distinguish the original author's views from inferences, and to retain the evidence number, path, and line number in citations.

🗂️ Automatic archiving and knowledge cards

tools/vault_curator.py Preview by default; after confirmation, use --apply to execute:

python3 tools/vault_curator.py archive "$VAULT_DIR"
python3 tools/vault_curator.py archive "$VAULT_DIR" --apply
python3 tools/vault_curator.py card "$VAULT_DIR" "10_Sources/x/example.md" --apply

archive Only process 00_Inbox/**/*.md, and file by platform, summary, and processed tags into 10_Sources/<platform>/ or 20_Processed/。
card generate 20_Processed/Cards/*.md, retaining the source, summary, key points, and tags.

🔌 MCP Server

scripts/mcp_server.py Provide search, read, and index tools for MCP-capable Agents. Use Python 3.10 or newer, and run from the full repository root:

python3 -m pip install -r knowledge-base-management/requirements-mcp.txt
python3 tools/mcp_smoke.py --json
VAULT_DIR=/path/to/your-vault python3 knowledge-base-management/scripts/mcp_server.py

The currently validated SDK is mcp==1.30.0。mcp_smoke.py Start a real stdio service with a temporary vault to verify the handshake, discovery of 6 tools, search, and read, without accessing your knowledge base.--help No SDK installation is required.

The installer for the full repository copies together vault_index.py、vault_curator.py、evidence_brief.py、import_document.py and the required common modules:

python3 tools/install_skill.py knowledge-base-management --dest ~/.codex/skills
python3 -m pip install -r ~/.codex/skills/knowledge-base-management/requirements-mcp.txt
VAULT_DIR=/path/to/your-vault python3 ~/.codex/skills/knowledge-base-management/scripts/mcp_server.py

The target skill directory must not exist; back up any existing installation first, or use a new installation directory. MCP preferentially loads the tools within the skill, and after installation does not depend on the original repository location.

The client configuration uses the real absolute paths for Python and the server:

{
  "mcpServers": {
    "chubby-kb": {
      "command": "/absolute/path/to/venv/bin/python",
      "args": ["/absolute/path/to/knowledge-base-management/scripts/mcp_server.py"],
      "env": { "VAULT_DIR": "/path/to/your-vault" }
    }
  }
}

The tool names remain 6:search_vault、semantic_search_vault、read_kb_note、list_recent_notes、reindex_vault、vault_index_stats. Before search, semantic retrieval, recent notes, and statistics, the index is automatically synchronized;read_kb_note reads the original text directly;reindex_vault() keeps the name unchanged and performs incremental synchronization.

The default database is the same as the CLI, located under the root directory at .chubby/index.sqlite; when customizing, set it in the client environment VAULT_INDEX_DB. The read path must be within VAULT_DIR .

The following are alloptional third-party/external tools, not provided with this repository. Paths (such as ~/graphrag-poc/、 ~/.hermes/...) are example locations on the author's local machine; please replace them with your own installation paths. Without them, it does not affect the core workflow in Sections 1 and 2 and scripts/vault_health_check.py。

knowledge base trio

Tool	Positioning	Use case
GBrain	Search	When you know what keyword to look for
GraphRAG	Discovery	When you do not know the keyword and want to discover hidden connections
LLM Wiki	Writing	Build an LLM-readable Wiki using the Karpathy mode
Understand-Anything	Visualization	Understand the structure of the entire library and discover implicit connections

GBrain Search Integration

Daily use：

alias gs='GBRAIN_SKIP_RECIPES=1 gbrain search'
gs "定价 品牌"
gs "奥德赛时期"

# 混合搜索
gbrain query "消费者心理剩余"

# 读全文（注意 slug 含 emoji 目录名）
gbrain get "wiki/📡 外部输入/播客/XXX/观点提取"

# 查看统计
gbrain stats

Configuration：DeepSeek downstream LLM（deepseek_api_key + deepseek_model: deepseek-chat + search.mode: balanced） Full manual：wiki/🧠 AI系统/GBrain-完整操作手册.md

GraphRAG Discovery

⚠️ Moved to a separate project: configure according to your actual installation path Flask API: localhost:8999 (default) Content: custom knowledge base + DeepSeek + BGE embeddings

Used to discover implicit associations and cross-domain connections in the knowledge base.

LLM Wiki (Karpathy Mode)

Karpathy's LLM Wiki mode: build an interlinked markdown knowledge base so the LLM can look it up directly during reasoning. Already integrated into the Obsidian vault.

Understand-Anything — Knowledge Graph Visualization

Installed as a standalone skill.

Usage：

# 必须指向 wiki 根目录，必须用 python3.11+
python3 parse-knowledge-base.py /path/to/your-vault/wiki

⚠️ Key pitfalls：

Python 3.11+ required
Must point to the wiki root directory (subdirectories without index.md will error)
Dashboard requires building the core package first
Requires GRAPH_DIR and ACCESS_TOKEN environment variables

4. Directory Organization (Organization)

Directory review and incremental cleanup

Execute when the user says "check whether the directory is reasonable":

Step 1: 全量目录扫描
  → find . -maxdepth 3 -type d | sort
  → 对每个 wiki 子目录统计 .md 文件数

Step 2: 逐项检查
  | 检查项 | 特征 | 严重性 |
  |--------|------|--------|
  | 旧版残留目录 | diff -rq 确认新旧版本内容不同 | 🔴 |
  | Legacy redirect 只剩 README | 目录只有一个 redirect 文件 | 🟡 |
  | 概念卡片散落根目录 | wiki/🧬 知识图谱/XXX.md 应在概念/下 | 🟡 |
  | cron 脚本写入旧路径 | grep -rn "旧目录" ~/.hermes/scripts/ | 🔴 |

Step 3: 分类执行 → 归档/删除/移动/更新引用

Step 4: 验证 → 零残留

Full-library audit and batch archiving

Step 1: 全量扫描目录结构和文件数
Step 2: 逐目录与最新管理办法对照
Step 3: 分级标记（🔴明确无用 → 🟡重复 → 🟠草稿 → 🔵过时）
Step 4: 批量 mv 到 wiki/🗄️ 归档/
Step 5: 修复残留引用（README/wiki/index/项目索引）

Archive directory structure：

wiki/🗄️ 归档/
├── 归档清单-YYYY-MM-DD.md
├── 旧版管理办法/
├── 旧版流程/
├── 旧版草稿/
├── 旧选题/
└── ...

Anti-patterns：

❌ Do not delete files under the archive directory
❌ After archiving, all index/README references must be updated in sync
❌ Files under 素材库/ are not archived
❌ Do not change file names when archiving
✅ Every major cleanup must create 归档清单-YYYY-MM-DD.md

5. Wiki Entity Management

Wiki entity format

Each team/project has a corresponding wiki/teams/ and under it a corresponding .md entity:

---
title: 团队名
created: YYYY-MM-DD
updated: YYYY-MM-DD
type: entity
tags: [team, ...]
sources: [50.团队/实际目录名]   # 重要：路径必须是实际存在的目录名
---

# 团队名

## Overview
## 文件结构（表格，列出子目录和数量）
## Related

The sources path must exactly match the actual directory name：

❌ sources: [50.团队/02.事业线] (old/incorrect)
✅ sources: [50.团队/事业线] (new/correct)

Handling abnormal files

Some .md files may be mistakenly treated as directory names (such as 千金-日报索引.md、OKR看板.md). How to check:

# 检查是否是文件被当作目录
for d in team_dir.iterdir():
    if d.is_file() and not d.name.startswith("."):
        print(f"⚠️ {d.name} 是文件却像目录")

Fix: use mv to move them into the corresponding directory.

6. Batch Repair Mode

Broken wikilink repair (podcast EP files)

Problem: wikilinks in podcast wiki files reference EP files in the 素材库, causing relative path resolution failures due to special characters in file names (quotes, emoji, backslashes).

Fix method：

# 1. 建索引: EP编号 → 实际文件路径
ep_index = {}
for f in 素材库_dir.iterdir():
    ep_match = re.match(r'(EP\d+)', f.name)
    if ep_match:
        ep_index[ep_match.group(1)] = f

# 2. 扫描 wiki 文件，用 EP 编号匹配实际文件，重建相对路径
for link in wikilinks:
    ep_match = re.search(r'(EP\d+)', target)
    if ep_match and ep_match.group(1) in ep_index:
        rel = os.path.relpath(ep_index[ep_id], md.parent)
        # 替换 wikilink

Batch add frontmatter (summary + source)

Problem: a large number of A+B outputs (viewpoint extraction/question chains) lack summary and source fields.

Fix method：

# summary: 从正文第一个 # 标题提取
title_match = re.search(r'^#\s+(.+)', body, re.MULTILINE)
summary = title_match.group(1).strip()[:100] if title_match else md.stem

# source: 从文件路径推断
if '观点提取' in md.name or '问题链' in md.name:
    source = '素材库/播客/给女孩的商业第一课'

⚠️ PITFALL: GraphRAG auto-generated entity files (🧬 知识图谱/GraphRAG*/实体/), about 260+, do not have summary; this is normal and does not need fixing.

Fix file names containing spaces

# 扫描
find wiki/ -name "* *" -type f

# 重命名
mv "MarkItDown 操作手册.md" "MarkItDown-操作手册.md"

Directory migration notice page format

When a directory is migrated as a whole, keep a README.md notice page at the old location:

---
title: "⚠️ 已迁移 - 目录名"
type: note
tags: [已废弃, 已迁移]
created: YYYY-MM-DD
summary: 已迁移到 ../../新位置。
source: "原系统名"
---

7. Business Line File Naming Conventions

Business line (50.团队/事业线/) file naming and Agent correspondence:

File name prefix	Agent	Should be placed in subdirectory
`O1-猎手` / `猎手-内容情报` / `猎手-渠道情报` / `猎手-技术调研` / `内容情报-给女孩的商业第一课`	Hunter	`猎手/`
`O2-笔神`	Pen God	`笔神/`
`O2-天眼`	Sky Eye	`天眼/`
`O3-管家`	Steward	`管家/`
`O4-极客`	Geek	`极客/`
`O4-画神`	Painting God	`画神/`
`千金-`	Thousand Gold	`千金/`

Special note：内容情报-给女孩的商业第一课-*.md These file names (87) are not series names; they are Hunter outputs and should be filed under the 猎手/ directory.

O3-CFO files: belong to 家庭线/CFO/, not a business line.

8. Thousand Gold Daily Report File Naming Conventions

Filename contains	Subdirectory
`总裁日报`	`总裁日报/`
`O1线`/`O2线`/`O3线`/`O4线`	`O线日报/`
`晨间简报`	`晨间简报/`
`浪漫-`	`浪漫/`
`系统-`	`系统/`
Other	`其他/`

Anti-patterns

❌ Do not delete any files under 素材库/
❌ Wiki pages must have a source field pointing to the original material
❌ Do not create isolated wiki pages with no inbound links (except index pages)
❌ Dry-run before cleanup
❌ Do not change filenames when archiving
✅ Every major cleanup must create 归档清单-YYYY-MM-DD.md

Related Skills

kb-ingest — Material ingestion (detailed process)
kb-audit — Health check and cleanup (detailed process)
kb-tools — GBrain/GraphRAG/LLM Wiki integration (detailed configuration)
chubby-knowledge-base-organization — Team file organization conventions
content-ab-processing — A+B dual-track processing
memory-obsidian-sync — Memory sync to Obsidian
understand / understand-knowledge — Knowledge graph visualization

About this skill

Core Architecture

Three-Layer Architecture

Vault Configuration

1. Material Ingestion (Ingest)

New Material Processing Pipeline

Material Grading (ABC Grading)

WeChat Official Account Article Processing

2. Health Check (Audit)

Automatic Check Items

Cleanup Criteria

File Naming Conventions

Migration Notice Page Format

3. Tool Integration (Tools)

🧠 Incremental Indexing and Unified Search

📥 Import Local Materials

📎 Verbatim Source Material Package

🗂️ Automatic archiving and knowledge cards

🔌 MCP Server

knowledge base trio

GBrain Search Integration

GraphRAG Discovery

LLM Wiki (Karpathy Mode)

Understand-Anything — Knowledge Graph Visualization

4. Directory Organization (Organization)

Directory review and incremental cleanup

Full-library audit and batch archiving

5. Wiki Entity Management

Wiki entity format

Handling abnormal files

6. Batch Repair Mode

Broken wikilink repair (podcast EP files)

Batch add frontmatter (summary + source)

Fix file names containing spaces

Directory migration notice page format

7. Business Line File Naming Conventions

8. Thousand Gold Daily Report File Naming Conventions

Anti-patterns

Related Skills