regex-vs-llm-structured-text

Structured Text Parsing: The Choice Between Regular Expressions (Regex) and Large Language Models (LLMs)

This is a practical decision framework for parsing structured text (quizzes, forms, invoices, documents). The core insight is: regular expressions (Regex) can deterministically handle 95-98% of cases at extremely low cost. Expensive LLM calls should be reserved for the remaining edge cases (Edge Cases).

When to Activate

• Parsing structured text with repetitive patterns (questions, forms, tables, etc.)
• Weighing whether to use regular expressions or LLMs in text extraction tasks
• Building hybrid pipelines (Hybrid Pipelines) that combine both approaches
• Optimizing the balance between cost and accuracy in text processing

Decision Framework

文本格式是否一致且重复？
├── 是 (>90% 遵循某种模式) → 从正则表达式（Regex）开始
│   ├── 正则表达式处理了 95%+ → 完成，无需 LLM
│   └── 正则表达式处理率 <95% → 仅针对边界情况添加 LLM
└── 否 (非格式化，高度多变) → 直接使用 LLM

Architecture Pattern

源文本（Source Text）
    │
    ▼
[正则解析器 (Regex Parser)] ─── 提取结构 (95-98% 准确率)
    │
    ▼
[文本清洗器 (Text Cleaner)] ─── 去除噪声 (标记、页码、人工痕迹)
    │
    ▼
[置信度评分器 (Confidence Scorer)] ─── 标记低置信度提取结果
    │
    ├── 高置信度 (≥0.95) → 直接输出
    │
    └── 低置信度 (<0.95) → [LLM 验证器 (LLM Validator)] → 输出

Implementation

1. Regex Parser (Handles the Vast Majority of Cases)

import re
from dataclasses import dataclass

@dataclass(frozen=True)
class ParsedItem:
    id: str
    text: str
    choices: tuple[str, ...]
    answer: str
    confidence: float = 1.0

def parse_structured_text(content: str) -> list[ParsedItem]:
    """使用正则表达式模式解析结构化文本。"""
    pattern = re.compile(
        r"(?P<id>\d+)\.\s*(?P<text>.+?)\n"
        r"(?P<choices>(?:[A-D]\..+?\n)+)"
        r"Answer:\s*(?P<answer>[A-D])",
        re.MULTILINE | re.DOTALL,
    )
    items = []
    for match in pattern.finditer(content):
        choices = tuple(
            c.strip() for c in re.findall(r"[A-D]\.\s*(.+)", match.group("choices"))
        )
        items.append(ParsedItem(
            id=match.group("id"),
            text=match.group("text").strip(),
            choices=choices,
            answer=match.group("answer"),
        ))
    return items

2. Confidence Scoring

Flag items that may require LLM review:

@dataclass(frozen=True)
class ConfidenceFlag:
    item_id: str
    score: float
    reasons: tuple[str, ...]

def score_confidence(item: ParsedItem) -> ConfidenceFlag:
    """对提取置信度进行评分并标记问题。"""
    reasons = []
    score = 1.0

    if len(item.choices) < 3:
        reasons.append("few_choices") # 选项过少
        score -= 0.3

    if not item.answer:
        reasons.append("missing_answer") # 缺失答案
        score -= 0.5

    if len(item.text) < 10:
        reasons.append("short_text") # 文本过短
        score -= 0.2

    return ConfidenceFlag(
        item_id=item.id,
        score=max(0.0, score),
        reasons=tuple(reasons),
    )

def identify_low_confidence(
    items: list[ParsedItem],
    threshold: float = 0.95,
) -> list[ConfidenceFlag]:
    """返回低于置信度阈值的项目。"""
    flags = [score_confidence(item) for item in items]
    return [f for f in flags if f.score < threshold]

3. LLM Validator (Handles Only Edge Cases)

def validate_with_llm(
    item: ParsedItem,
    original_text: str,
    client,
) -> ParsedItem:
    """使用 LLM 修复低置信度的提取结果。"""
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",  # 使用最便宜的模型进行验证
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": (
                f"Extract the question, choices, and answer from this text.\n\n"
                f"Text: {original_text}\n\n"
                f"Current extraction: {item}\n\n"
                f"Return corrected JSON if needed, or 'CORRECT' if accurate."
            ),
        }],
    )
    # 解析 LLM 响应并返回修正后的项目...
    return corrected_item

4. Hybrid Pipeline

def process_document(
    content: str,
    *,
    llm_client=None,
    confidence_threshold: float = 0.95,
) -> list[ParsedItem]:
    """完整流水线：正则提取 -> 置信度检查 -> 针对边界情况调用 LLM。"""
    # 步骤 1: 正则提取 (处理 95-98% 的情况)
    items = parse_structured_text(content)

    # 步骤 2: 置信度评分
    low_confidence = identify_low_confidence(items, confidence_threshold)

    if not low_confidence or llm_client is None:
        return items

    # 步骤 3: LLM 验证 (仅针对被标记的项目)
    low_conf_ids = {f.item_id for f in low_confidence}
    result = []
    for item in items:
        if item.id in low_conf_ids:
            result.append(validate_with_llm(item, content, llm_client))
        else:
            result.append(item)

    return result

Real-World Metrics

From a production quiz parsing pipeline (410 items):

Metric	Value
Regex Success Rate	98.0%
Number of Low-Confidence Items	8 (2.0%)
Number of LLM Calls Required	~5
Cost Savings Compared to an All-LLM Approach	~95%
Test Coverage	93%

Best Practices

• Start with regular expressions — even an imperfect regex gives you a baseline to improve from.
• Use confidence scoring — programmatically identify what needs LLM assistance.
• Use the cheapest LLM for validation — Haiku-level models are usually sufficient.
• Never mutate the original parsed items — return new instances in the cleaning/validation step.
• Test-driven development (TDD) works very well — write tests for known patterns first, then handle edge cases.
• Log metrics — log regex success rates, LLM call counts, etc., to track pipeline health.

Anti-Patterns to Avoid

• Sending all text to an LLM even when regex can handle over 95% of cases (both expensive and slow).
• Using regex on unstructured, highly variable text (LLMs perform better in this case).
• Skipping confidence scoring and hoping the regex "just works."
• Directly modifying parsed objects in the cleaning/validation step.
• Not testing edge cases (malformed input, missing fields, encoding issues, etc.).

When to Use

• Quiz/exam question parsing
• Form data extraction
• Invoice/receipt processing
• Document structure parsing (headings, sections, tables)
• Any structured text processing with repetitive patterns that is cost-sensitive