doc-importer

Document Importer (Doc Importer)

Batch import local documents (Obsidian, SiYuan Notes, WeChat Official Account HTML, the Downloads directory, or any directory) into WPS Notes.

---

Quick Start

The note writing for this skill uniformly goes through the Notes Agent tool; if you need to understand the batch scanning and conversion logic, you can first read the scripts bundled with this Skill:

参考实现：
- read_file("scripts/scan_docs.py")        # 目录扫描与来源识别
- read_file("scripts/import_to_wps.py")    # 转换与导入主流程

Actual writing:

• create_note
• get_note_outline
• batch_edit
• insert_image
• read_note / read_blocks
• search_notes
• sync_note

---

## 支持的文档来源

| 来源 | 说明 | 典型目录结构 |
|------|------|------------|
| **Obsidian Vault** | 扫描 .md 文件，保留 wiki 链接、Callout、标签 | `Vault/笔记名.md` + `attachments/` |
| **思源笔记** | 扫描 .sy 文件（JSON格式），提取 Block Tree | `SiYuan/data/笔记本/文档.sy` |
| **微信公众号** | 解析 `原文.html`（含富文本格式、内联样式） | `文章名/原文.html` + `图片/` |
| **任意目录** | 用户指定路径，递归扫描子目录 | 任意 |

---

## 支持的文件格式

| 格式 | 转换方式 | 图片处理 | 富文本保留 |
|------|----------|----------|-----------|
| `.html` | BeautifulSoup 解析内联样式 | 本地图片 base64 | ✅ 颜色/粗体/标题 |
| `.md` / `.markdown` | 直接解析，转 WPS XML | 本地图片 base64 | 基本格式 |
| `.pdf` | pdfplumber 提取文本 + pdfimages 提取图片 | 提取嵌入图片 | 标题推断 |
| `.docx` | pandoc 转 markdown，提取 word/media/ | 解包提取 | 基本格式 |
| `.pptx` | markitdown 提取文本 + 解包媒体 | 解包提取 | 幻灯片结构 |
| `.xlsx` | pandas 读取，转 WPS table | 不含图片 | 表格结构 |
| `.txt` | 直接读取 | 不含图片 | 无 |
| `.sy` | JSON 解析思源 Block Tree | 提取 assets 图片 | 全部 |

---

## WPS API 关键限制（必读）

在实际导入前，必须了解以下 WPS 笔记 API 的重要限制，否则会导致内容丢失或图片插入失败。

### 1. `get_note_outline` 默认只返回 100 个 block

`get_note_outline` 默认按分页视图返回 block，`blocks` 数组可能只覆盖前一部分内容，但 `block_count` 会返回真实总数。

**影响**：大文章（> 100 个 block）如果只靠首屏大纲查占位符，只能找到前一部分 block，后半段图片会丢失。

**解决方案**：先用 `get_note_outline` 拿首批 block，再用 `read_blocks` 续读：

```python
def get_all_blocks(note_id):
    """翻页获取笔记全部 blocks"""
    # 第一页用 get_note_outline（有 preview 字段）
    r = get_note_outline(note_id=note_id)
    data = r.get('data', {})
    total = data.get('block_count', 0)
    blocks = list(data.get('blocks', []))
    last_id = blocks[-1]['id'] if blocks else None

    # 超过首批范围时用 read_blocks 续读
    while len(blocks) < total and last_id:
        r2 = read_blocks(note_id=note_id, block_id=last_id, after=100)
        new_blocks = (r2.get('data') or {}).get('blocks', [])
        if not new_blocks:
            break
        blocks.extend(new_blocks)
        last_id = new_blocks[-1]['id']

    return blocks

get_note_outline The returned block often carries the preview field;read_blocks is better suited for getting the full XML content.

---

2. insert_image In older versions of WPS, it can only be inserted into the foreground note

Older versions (< 0.1.4)：insert_image images fileID are bound to the note open in the current WPS client UI; even if the note_id parameter is passed, the images will still be incorrectly associated with the foreground note.

New versions (>= 0.1.4): fixed,insert_image can insert images into any note in the background directly, with no need to switch to the foreground.

Handling Strategy：

• First execute according to the background image insertion capability; if the image is inserted in the wrong place, then fall back to "switch to the target note and then insert the image"
• If the host or client still exhibits the old-version behavior, clearly prompt the user to upgrade or manually switch to the target note
• After inserting images, be sure to read back for verification; do not assume success

---

3. list_notes When large data is returned, proactively narrow the scope

list_notes When the data returned at one time is too large, it is not suitable for full-text deduplication or full-scale scan judgments.

Solution：

• For deduplication, prioritize using search_notes to search by title keywords
• When browsing lists, proactively paginate or narrow the filtering scope
• Do not treat a single large-scope list_notes as the sole source of truth

---

4. Anchor invalidation causes silent content loss

When calling batch_edit consecutively to insert content, WPS may reindex internally, causing the anchor_id returned by the previous operation to become invalid. If this is not handled, subsequent content will be silently skipped, causing the latter half of the article content to be lost.

Retry logic must be implemented：

def do_insert(note_id, content, get_anchor_fn, max_retries=4):
    """带重试的内容插入，自动刷新 anchor"""
    anchor = get_anchor_fn()
    for attempt in range(max_retries):
        if attempt > 0:
            time.sleep(1.5 * attempt)
            anchor = get_anchor_fn()  # 重新获取最新 anchor
        res = batch_edit(note_id, [{
            'op': 'insert', 'anchor_id': anchor,
            'position': 'after', 'content': content
        }])
        if res.get('ok') is not False:
            anchor = get_anchor_fn()
            return True
    return False  # 4次都失败才放弃

Get the latest anchor (the last block)：

def get_last_block_id(note_id):
    r = get_note_outline(note_id=note_id)
    blocks = (r.get('data') or {}).get('blocks', [])
    return blocks[-1]['id'] if blocks else None

---

5. Stable parameters for batch writing

When writing a large amount of content,BATCH_SIZE(the number of blocks per batch_edit ), it is recommended to set it to 4; too large a value increases the probability of anchor invalidation.

BATCH_SIZE = 4  # 不要超过 8，否则容易出现 anchor 失效

---

Complete Workflow

Step 1: Determine the scan directory

First determine the source type based on the directory/file clues the user has already given; if insufficient, then ask_user to fill in the minimum necessary information:

优先识别这些常见来源线索：
- Obsidian Vault：`.md` 文件 + `attachments/` 或同类附件目录
- 思源笔记：`.sy` 文件 + `data/assets/`
- 微信公众号存档：`原文.html` + 图片目录 + `meta.json`
- 下载目录混合导入：PDF / DOCX / PPTX / XLSX / Markdown / HTML 混合存在

---

Step 2: Scan the document list

如需对照扫描规则，用 `read_file("scripts/scan_docs.py")` 查看来源识别和输出字段约定；
整理后的清单至少要包含路径、标题、时间、格式、图片数和预计 block 数。

Output in JSON format:

{
  "source_type": "wechat_mp",
  "root_path": "/Users/xxx/Documents/articles",
  "files": [
    {
      "path": "/Users/xxx/Documents/articles/文章名/原文.html",
      "rel_path": "文章名/原文.html",
      "title": "文章标题",
      "publish_time": "2025-04-21 18:30",
      "size_bytes": 204800,
      "modified": "2025-04-22T10:00:00",
      "format": "html",
      "estimated_images": 12,
      "estimated_blocks": 180
    }
  ],
  "total": 71,
  "formats": {"html": 71}
}

estimated_blocks helps predict in advance whether it will exceed 100 blocks, and prompt the user in advance.

---

Step 3: Display the file list and ask for a selection

扫描到 71 个文件：
  - HTML: 71 个

文件列表：
 1. AutoGLM 发布之后，如今国产大模型终于长出了手。  (2025-03-31, 12张图)
 2. 你可能看不懂扣子空间为什么重要…                 (2025-04-21, 17张图)
 ...（超过20个时截断，告知总数）

请问你想如何导入？
 [A] 全部导入（71个文件）
 [B] 手动选择（输入文件编号，如：1,3,5-10）
 [S] 跳过已有标题的笔记（根据笔记标题去重）

---

Step 4: Deduplication detection

Before importing, check by title whether a note with the same name already exists in WPS Notes.

Note: Do not treat list_notes as the main entry point for deduplication; instead use search_notes to search by title keywords:

def check_exists(title):
    r = search_notes(keyword=title[:20], limit=5)
    notes = (r.get('data') or {}).get('notes', [])
    return next((n for n in notes if n['title'] == title), None)

When duplicates are found, ask the user:

发现以下笔记在 WPS 中已存在：
 - 《AutoGLM 发布之后…》（最后更新：2025-05-01）

如何处理？
 [O] 覆盖  [S] 跳过  [A] 追加  [RA] 对所有冲突应用相同策略

---

Step 5: Convert and import

Recommended workflow (using HTML rich text as an example):

# 1. 解析 HTML，提取内容段落和图片
segments = html_to_segments(html_path, img_dir)
# segments 格式：[('xml', '<p>文字</p>'), ('img', Path('图片/image_001.jpg')), ...]

# 2. 创建笔记
note_id = create_note(title)

# 3. 写入标题行 + meta 行（时间、标签）
write_header(note_id, title, publish_time, tag)

# 4. 批量写入正文（图片先插占位符）
write_content_with_placeholders(note_id, segments)

# 5. 翻页查找所有占位符 block_id（用 get_all_blocks）
ph_map = find_placeholders(note_id)

# 6. 逐个插入真实图片，替换占位符
for idx, img_path in img_list:
    insert_image(note_id, ph_map[idx]['block_id'], img_path)
    delete_placeholder(note_id, ph_map[idx]['block_id'])

Meta row format(adjust flexibly according to source type):

# 微信公众号：简洁的时间 + 标签
f'<p>{publish_time} | <tag id="{tag_id}">#推文</tag></p>'

# 通用文档：完整 meta blockquote
"""
<blockquote>
  <p>📄 <strong>来源</strong>：{rel_path}</p>
  <p>🕒 <strong>修改时间</strong>：{modified_time}</p>
  <p>🔄 <strong>导入时间</strong>：{import_time}</p>
</blockquote>
"""

---

Step 6: Progress report

[3/71] AutoGLM 发布之后…
  解析完成: 95 段文字, 12 张图片
  ✓ 创建笔记: 501435173515
  ✓ 文字写入完成
  ✓ 图片插入: 12/12
  用时: 8.3s

进度：████████░░░░  42% (30/71)  预计剩余 ~12 分钟

---

Conversion script reference (when you need to look at the implementation)

优先阅读这些文件来理解不同来源的导入逻辑：
- read_file("scripts/import_to_wps.py")  # 主导入流程、去重、图片插入
- read_file("scripts/scan_docs.py")      # 目录扫描、来源判断、筛选
- read_file("references/conversion-guide.md")  # 格式映射细节

---

Step-by-step writing to the note-taking tool (when manual takeover is needed)

# 1. 创建笔记
create_note(title="文档标题")

# 2. 获取初始 block ID
get_note_outline(note_id=note_id)

# 3. 写入内容（分批，每批 4 个 block）
batch_edit(note_id=note_id, operations=[
    {"op": "replace", "block_id": first_block_id, "content": "<h1>标题</h1>"},
    {"op": "insert", "anchor_id": first_block_id, "position": "after",
     "content": "<p>2025-04-21 18:30 | <tag>#推文</tag></p>"},
])

# 4. 插入图片（新版 WPS 支持后台插图）
insert_image(note_id=note_id, anchor_id=placeholder_block_id,
             position="before", src="data:image/jpeg;base64,...")

---

Special handling for each source

WeChat Official Account HTML (原文.html）

WeChat Official Account articles use inline CSS styles to express rich text, and the format needs to be parsed from HTML:

Directory structure:

文章目录/
  原文.html        ← 完整 HTML，含内联样式
  meta.json        ← {"title": "...", "publish_time": "2025-04-21 18:30", "url": "..."}
  图片/            ← 本地图片文件
    image_001.jpg
    image_002.jpg

HTML parsing key points:

• The body is inside the #js_content container
• Images are in the <img data-src="https://..."> attribute (not src）
• Headings are inferred through the span of font-size >= 18px span inference
• Colors are extracted through the color: rgb(...) style of the span and need to be mapped to WPS preset colors
• Bold is identified through font-weight: 700 or bold recognition

Inline styles → WPS XML mapping:

# font-size >= 18px → <h2>
# font-weight: 700|bold → <strong>
# font-style: italic → <em>
# color: rgb(R,G,B) → <span fontColor="#WPS预设色">（需颜色映射）

For detailed color mapping rules, see references/conversion-guide.md Section 10.

---

Obsidian Vault

1. Wiki links：[[文件名]] → plain text;[[文件名|显示名]] → 显示名
2. Tags：#标签名 → <tag>#标签名</tag>
3. Callouts：> [!NOTE] → WPS <highlightBlock>
4. Frontmatter: YAML front matter is extracted as meta information
5. Image paths: first look in the same directory → then look in attachments/ → then look in the Vault root directory

---

SiYuan Note (SiYuan)

• .sy files are in JSON format as a Block Tree
• The image path format is assets/xxx.png, the actual file is in <工作空间>/data/assets/
• For detailed node type mapping, see references/conversion-guide.md Section 6

---

Troubleshooting

get_note_outline only returns 100 blocks, and the subsequent content is lost

Use the pagination solution, see item 1 of "WPS API Key Limitations".

After the image is inserted, clicking it shows a loading failure

Older WPS (< 0.1.4)insert_image fileID is bound to the foreground note; upgrading to the latest version can solve this.

insert_image reports IMAGE_FETCH_FAILED

• Check the base64 data URI format: it must be data:image/jpeg;base64,<数据>(it cannot be data:application/octet-stream）
• For large images, prefer a stable URL; if only base64 can be passed, make sure what is passed is a complete data URI

anchor failure causes content truncation

Implement a 4-retry logic, reacquiring after each failure last_block_id, see item 4 of "WPS API Key Limitations".

list_notes / search_notes returns too many results

Narrow the filtering scope or actively paginate, see item 3 of "WPS API Key Limitations".

pandoc is not installed

brew install pandoc  # macOS

pdfplumber is not installed

pip3 install pdfplumber

Scanned PDF has no text (OCR required)

pip3 install pytesseract pdf2image
brew install tesseract

---

Dependency list

Tool	Purpose	Installation
Note Agent tool (create_note / get_note_outline / batch_edit / insert_image / read_note / read_blocks / search_notes / sync_note）	WPS note writing, deduplication, and read-back (required)	provided by the host
beautifulsoup4	HTML parsing (Official Account/web pages)	pip3 install beautifulsoup4
lxml	HTML parsing acceleration	pip3 install lxml
pandoc	DOCX → Markdown	brew install pandoc
pdfplumber	PDF text/table extraction	pip3 install pdfplumber
pypdf	PDF image extraction alternative	pip3 install pypdf
markitdown	PPTX → Markdown	pip3 install "markitdown[pptx]"
pandas	XLSX reading	pip3 install pandas openpyxl
pillow	Image processing/base64 conversion	pip3 install pillow
python-frontmatter	YAML Frontmatter parsing	pip3 install python-frontmatter

For detailed conversion logic, see references/conversion-guide.md。