pdf-bib-import

Batch import PDF citations into Feishu Bitable

Core principles (must be followed)

Cost control: Data extraction must be done with scripts; using AI to read PDFs one by one is prohibited—each AI read consumes a large number of tokens.

Correct process:Write a script to extract → manual/script proofreading → script batch write to Feishu, AI is only responsible for generating scripts and configuring the Feishu structure.

---

Step 1: Script extraction of PDF citations

1.1 Generate extraction script

In /tmp generate extract_bib.py, using pdftotext（/opt/homebrew/bin/pdftotext) batch extract:

#!/usr/bin/env python3
"""
extract_bib.py — 从 PDF 目录批量提取题录，输出 bib_data.json
用法：python3 extract_bib.py <pdf_dir>
"""
import os, sys, json, subprocess, re

def run(cmd):
    return subprocess.run(cmd, capture_output=True, text=True).stdout.strip()

def extract_pdf_meta(path):
    info = run(["/opt/homebrew/bin/pdfinfo", path])
    meta = {}
    for line in info.splitlines():
        if ":" in line:
            k, _, v = line.partition(":")
            meta[k.strip()] = v.strip()
    text = run(["/opt/homebrew/bin/pdftotext", "-l", "2", path, "-"])
    return meta, text

def guess_fields(meta, text):
    """从 pdfinfo + 前两页文字中猜测题录字段，返回 dict。"""
    lines = [l.strip() for l in text.splitlines() if l.strip()]
    return {
        "论文题目": meta.get("Title", ""),
        "作者":     meta.get("Author", ""),
        "发表年份": None,
        "期刊名称": "",
        "卷期页码": "",
        "DOI":      "",
        "摘要":     "",
        "_raw_text_preview": "\n".join(lines[:40]),  # 供人工核查
    }

def main():
    pdf_dir = sys.argv[1] if len(sys.argv) > 1 else "."
    results = []
    for fname in sorted(os.listdir(pdf_dir)):
        if not fname.lower().endswith(".pdf"):
            continue
        fpath = os.path.join(pdf_dir, fname)
        meta, text = extract_pdf_meta(fpath)
        rec = guess_fields(meta, text)
        rec["_filename"] = fname
        results.append(rec)
        print(f"  ✓ {fname}")

    out = "/tmp/bib_data.json"
    with open(out, "w", encoding="utf-8") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    print(f"\n已写出 {len(results)} 条记录 → {out}")
    print("请检查 bib_data.json，补全缺失字段后再执行 Step 3 导入。")

if __name__ == "__main__":
    main()

Run:

python3 /tmp/extract_bib.py "/path/to/pdf_folder"

1.2 Proofread JSON (key step)

pdftotext can extract text from machine-readable PDFs, but some PDFs:

• Scanned copies: Text is empty, needs manual filling
• Metadata is empty: Need to determine from the _raw_text_preview field and manually complete

Open /tmp/bib_data.json, check and complete each field, delete _raw_text_preview and _filename。

---

Step 2: Create Feishu Base structure

Prerequisites: First read ../lark-shared/SKILL.md to understand the authentication rules.

2.1 Create Base

lark-cli base +base-create --name "论文题录数据库"
# 记录返回的 base_token

2.2 Get the default table and rename it

lark-cli base +table-list --base-token <BASE_TOKEN>
lark-cli base +table-update --base-token <BASE_TOKEN> --table-id <TABLE_ID> --json '{"name":"论文题录"}'

2.3 Configure fields

First rename the default "Text" field, then delete redundant default fields, and finally create the required fields:

# 查看现有字段（记录各字段 id）
lark-cli base +field-list --base-token <BASE_TOKEN> --table-id <TABLE_ID>

# 将默认文本字段重命名为"论文题目"
lark-cli base +field-update --base-token <BASE_TOKEN> --table-id <TABLE_ID> \
  --field-id <TEXT_FIELD_ID> --json '{"type":"text","name":"论文题目","style":{"type":"plain"}}'

# 删除多余默认字段（日期、附件、单选）
lark-cli base +field-delete --base-token <BASE_TOKEN> --table-id <TABLE_ID> --field-id <ID> --yes

# 创建其余字段（串行执行）
cd /tmp
for field in \
  '{"type":"text","name":"作者","style":{"type":"plain"}}' \
  '{"type":"number","name":"发表年份"}' \
  '{"type":"text","name":"期刊名称","style":{"type":"plain"}}' \
  '{"type":"text","name":"卷期页码","style":{"type":"plain"}}' \
  '{"type":"text","name":"DOI","style":{"type":"plain"}}' \
  '{"type":"text","name":"摘要","style":{"type":"plain"}}'; do
  echo "$field" > field.json
  lark-cli base +field-create --base-token <BASE_TOKEN> --table-id <TABLE_ID> --json @field.json
  sleep 0.3
done

---

Step 3: Script batch write records

3.1 Generate import script

Generate /tmp/import_to_lark.sh, read records from /tmp/bib_data.json and write them one by one:

cat << 'PYEOF' > /tmp/import_to_lark.py
#!/usr/bin/env python3
"""
import_to_lark.py — 从 bib_data.json 批量写入飞书 Base
用法：python3 import_to_lark.py <BASE_TOKEN> <TABLE_ID> [bib_data.json]
"""
import json, sys, subprocess, time, os

def upsert(base_token, table_id, record):
    tmp = "/tmp/_rec.json"
    # 去除空值字段（避免写入空字符串覆盖）
    payload = {k: v for k, v in record.items()
               if not k.startswith("_") and v not in (None, "", 0)}
    # 年份是数字，0 代表未知，保持过滤
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump(payload, f, ensure_ascii=False)
    result = subprocess.run(
        ["lark-cli", "base", "+record-upsert",
         "--base-token", base_token,
         "--table-id", table_id,
         "--json", "@_rec.json"],
        capture_output=True, text=True, cwd="/tmp"
    )
    return json.loads(result.stdout)

def main():
    base_token = sys.argv[1]
    table_id   = sys.argv[2]
    data_file  = sys.argv[3] if len(sys.argv) > 3 else "/tmp/bib_data.json"

    with open(data_file, encoding="utf-8") as f:
        records = json.load(f)

    ok, fail = 0, 0
    for i, rec in enumerate(records, 1):
        title = rec.get("论文题目", rec.get("_filename", f"record-{i}"))
        resp = upsert(base_token, table_id, rec)
        if resp.get("ok"):
            ok += 1
            print(f"  [{i}/{len(records)}] ✓ {title[:60]}")
        else:
            fail += 1
            print(f"  [{i}/{len(records)}] ✗ {title[:60]}: {resp}")
        time.sleep(0.5)   # 避免写入冲突

    print(f"\n完成：{ok} 成功 / {fail} 失败")

if __name__ == "__main__":
    main()
PYEOF

python3 /tmp/import_to_lark.py <BASE_TOKEN> <TABLE_ID> /tmp/bib_data.json

---

Common issues

Issue	Solution
pdftotext extraction is empty (scanned copy)	Manually fill in bib_data.json, or use an OCR tool
PDF metadata is garbled / incorrect	Ignore meta, manually extract from _raw_text_preview Manual extraction
lark-cli --json @file reports a path error	Must cd /tmp before executing, use relative path @_rec.json
Write conflict 1254291	Increase time.sleep() to 1 second
Field does not exist 1254045	Check the field name spelling, consistent with +field-list returned result

Cost control tips

• Do not use AI to read PDFs one by one: 49 PDFs extracted with AI ≈ $10, with a script ≈ $0
• Data extraction → manual proofreading → script write, AI only generates scripts
• When encountering fields that pdftotext cannot extract, concentrate on manually completing the JSON, then import all at once