doc-extract-engine

Document Extraction Engine

Perform end-to-end structured information extraction from PDF documents via a remote API.

Environment Configuration

Before use, make sure to set the environment variables:

export EXTRACT_API_URL="http://39.104.68.74:8082"   # API 地址（默认值）
export EXTRACT_USERNAME="your_username"
export EXTRACT_PASSWORD="your_password"
# 或直接提供 token（跳过登录）
export EXTRACT_TOKEN="eyJhbGciOiJIUzI1NiIs..."

Dependencies:pip install httpx

Core Workflow

注册(首次) → 登录 → 上传PDF → Schema对话(AI设计) → 创建Session → 提取(SSE) → 对话修正 → 归档

Usage

Method 1: Python programmatic invocation (recommended, used directly by Claude)

Add the scripts/ directory to sys.path, and import client directly:

import sys
sys.path.insert(0, "<skill_path>/scripts")
from client import ExtractEngineClient

client = ExtractEngineClient("http://39.104.68.74:8082")

# 注册（首次使用，邮箱必须用合法域名如 @example.com，不能用 @xxx.local）
try:
    client.register("myuser", "mypassword", "myuser@example.com")
except Exception:
    pass  # 已注册则跳过

# 登录
client.login("myuser", "mypassword")

# 上传
docs = client.upload_documents(["doc1.pdf"])
doc_ids = [d["document_id"] for d in docs]

# Schema 对话（可多轮）
conv_id = client.create_schema_conversation(doc_ids)
result = client.chat_schema(conv_id, "提取论文标题和作者")

# ⚠️ schema 可能在 SSE 流中未返回，需 fallback 查询
schema = result.get("schema")
if not schema:
    conv_state = client.get_schema_conversation(conv_id)
    schema = conv_state.get("current_schema")

# 创建 Session 并提取
session_id = client.create_session("我的任务", schema, doc_ids)
result = client.extract(session_id, doc_ids)
batch_id = result["batch_id"]

# 修正（可选，可多轮）
client.chat_correct(batch_id, "作者列表不完整")

# 归档
client.end_batch(batch_id)

Method 2: CLI script (suitable for scenarios where environment variables are already configured)

# 完整流程：上传 → Schema → 提取 → 归档
python scripts/extract_flow.py full \
  --files doc1.pdf doc2.pdf \
  --schema-prompt "提取论文标题、作者、摘要和关键词"

# 单步操作
python scripts/extract_flow.py upload --files doc1.pdf
python scripts/extract_flow.py extract --session-id 1 --doc-ids abc123
python scripts/extract_flow.py correct --batch-id 1 --message "作者不完整"
python scripts/extract_flow.py archive --batch-id 1
python scripts/extract_flow.py history --session-id 1

Workflow Decision Tree

用户请求提取文档信息
  │
  ├─ 有现成 Session？
  │   ├─ 是 → 直接 extract（复用 schema）
  │   └─ 否 → 需要先设计 Schema
  │
  ├─ 有明确的 Schema？
  │   ├─ 是 → 跳过 Schema 对话，直接 create_session
  │   └─ 否 → 上传文档 → Schema 对话（AI 辅助设计）
  │
  ├─ 提取结果需要修正？
  │   ├─ 是 → chat_correct 多轮对话
  │   └─ 否 → 直接归档
  │
  └─ 需要查看历史？
      └─ get_history(session_id)

SSE Streaming Interface Handling

The extraction and dialogue interfaces return SSE streams. The client already encapsulates streaming consumption. Key events:

batch_created → record batch_id, needed for subsequent correction and archiving
result_update → extraction result, thecontent field is the result array
schema_updated → the Schema generated in the Schema dialogue
error → error message, the client automatically throws an exception
done → end of stream

Custom event handling via the on_event callback:

def my_handler(event):
    if event["type"] == "result_update":
        save_to_file(event["content"])

client.extract(session_id, doc_ids, on_event=my_handler)

Key Constraints and Pitfall Experience

Each upload is limited to 1-3 PDF files
The Schema dialogue supports multiple rounds until the Schema is satisfactory
Dialogue correction supports multiple rounds until the result is satisfactory
After archiving, the system automatically triggers background tasks for experience summarization and code summarization
All non-SSE interfaces return a unified envelope format {code, success, message, data}

Known Issues

Strict registration email validation: does not accept @xxx.local and other reserved domains; use @example.com and other legitimate domains
Schema fallback：chat_schema() In the SSE stream of schema_updated the event may not carry the schema; be sure to use get_schema_conversation() as a fallback to obtain it
SSE timeout: the extraction and dialogue interfaces may take a long time; the client _consume_sse has been set to timeout=None, and will not time out and disconnect

Reference Materials

API interface details: see references/api_guide.md