content-hash-cache-pattern

Content-Hash File Cache Pattern

Use SHA-256 content hashes (rather than file paths) as cache keys to cache expensive file processing results (such as PDF parsing, text extraction, and image analysis). Unlike path-based caching, this approach remains effective after files are moved/renamed, and automatically invalidates when content changes.

When to Activate

• Building file processing pipelines, such as PDF, image, or text extraction
• Processing is expensive and the same files need to be processed repeatedly
• Need to provide --cache/--no-cache command-line options
• Want to add caching to existing pure functions without modifying them

Core Pattern

1. Content-hash-based cache keys

Use file content (rather than path) as the cache key:

import hashlib
from pathlib import Path

_HASH_CHUNK_SIZE = 65536  # 大文件采用 64KB 分块读取

def compute_file_hash(path: Path) -> str:
    """计算文件内容的 SHA-256 哈希值（针对大文件进行分块）。"""
    if not path.is_file():
        raise FileNotFoundError(f"File not found: {path}")
    sha256 = hashlib.sha256()
    with open(path, "rb") as f:
        while True:
            chunk = f.read(_HASH_CHUNK_SIZE)
            if not chunk:
                break
            sha256.update(chunk)
    return sha256.hexdigest()

Why use content hashes? File rename/move = cache hit. Content change = automatic invalidation. No index file needed.

2. Frozen Dataclass for cache entries

from dataclasses import dataclass

@dataclass(frozen=True, slots=True)
class CacheEntry:
    file_hash: str
    source_path: str
    document: ExtractedDocument  # 缓存的结果内容

3. File-based cache storage

Each cache entry is stored as {hash}.json — O(1) lookup via hash, with no index file needed.

import json
from typing import Any

def write_cache(cache_dir: Path, entry: CacheEntry) -> None:
    cache_dir.mkdir(parents=True, exist_ok=True)
    cache_file = cache_dir / f"{entry.file_hash}.json"
    data = serialize_entry(entry)
    cache_file.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")

def read_cache(cache_dir: Path, file_hash: str) -> CacheEntry | None:
    cache_file = cache_dir / f"{file_hash}.json"
    if not cache_file.is_file():
        return None
    try:
        raw = cache_file.read_text(encoding="utf-8")
        data = json.loads(raw)
        return deserialize_entry(data)
    except (json.JSONDecodeError, ValueError, KeyError):
        return None  # 将损坏的缓存视为未命中

4. Service layer wrapper (satisfies the Single Responsibility Principle, SRP)

Keep processing functions pure. Add caching logic as a separate service layer.

def extract_with_cache(
    file_path: Path,
    *,
    cache_enabled: bool = True,
    cache_dir: Path = Path(".cache"),
) -> ExtractedDocument:
    """服务层逻辑：检查缓存 -> 提取内容 -> 写入缓存。"""
    if not cache_enabled:
        return extract_text(file_path)  # 纯函数，不感知缓存逻辑

    file_hash = compute_file_hash(file_path)

    # 检查缓存
    cached = read_cache(cache_dir, file_hash)
    if cached is not None:
        logger.info("Cache hit: %s (hash=%s)", file_path.name, file_hash[:12])
        return cached.document

    # 缓存未命中 -> 提取 -> 存储
    logger.info("Cache miss: %s (hash=%s)", file_path.name, file_hash[:12])
    doc = extract_text(file_path)
    entry = CacheEntry(file_hash=file_hash, source_path=str(file_path), document=doc)
    write_cache(cache_dir, entry)
    return doc

Key Design Decisions

Decision	Rationale
SHA-256 content hash	Path-independent, automatically invalidates when content changes
{hash}.json File naming	O(1) lookup speed, no index file to maintain
Service layer wrapper	Satisfies the Single Responsibility Principle (SRP): extraction logic stays pure, caching is a separate concern
Manual JSON serialization	Full control over serialization of frozen dataclasses
Corruption returns None	Graceful degradation, reprocessed on the next run
cache_dir.mkdir(parents=True)	Lazily create directories on first write

Best Practices

• Hash content rather than paths — paths change, content identity does not.
• Hash large files in chunks — avoid loading the entire file into memory.
• Keep processing functions pure — they should not know anything about caching.
• Log cache hits/misses — use truncated hash values for debugging.
• Handle cache corruption gracefully — treat invalid cache entries as misses, and never crash because of them.

Anti-Patterns to Avoid

# 错误做法：基于路径的缓存（文件移动/重命名后失效）
cache = {"/path/to/file.pdf": result}

# 错误做法：在处理函数内部添加缓存逻辑（违反 SRP）
def extract_text(path, *, cache_enabled=False, cache_dir=None):
    if cache_enabled:  # 该函数现在有了双重职责
        ...

# 错误做法：对嵌套的冻结数据类直接使用 dataclasses.asdict()
# (在处理复杂的嵌套类型时可能会出现问题)
data = dataclasses.asdict(entry)  # 推荐使用手动序列化

Applicable Scenarios

• File processing pipelines (PDF parsing, OCR, text extraction, image analysis)
• Command-line tools (CLI) that benefit from --cache/--no-cache options
• Batch tasks where the same files appear across different runs
• Adding caching to existing pure functions without modifying them

Inapplicable Scenarios

• Data that must remain absolutely real-time (real-time data streams)
• Cache entries are extremely large in size (in this case, streaming should be considered)
• The result depends on parameters other than the file content (for example, different extraction configurations)