OpenAI Privacy Filter - OpenAI's open-source privacy filtering model

What is OpenAI Privacy Filter

OpenAI Privacy Filter is OpenAI's open-source privacy filtering model, designed specifically to detect and redact personally identifiable information (PII) in text. The model is based on gpt-oss The architecture is adapted into a bidirectional token classifier, with 1.5 billion total parameters and 50 million active parameters, supporting an ultra-long context of 128,000 tokens, and can run locally in a browser or notebook without uploading sensitive data to the cloud. The model achieves a 96% F1 score on the PII-Masking-300k benchmark (97.43% after correction).

Main features of OpenAI Privacy Filter

• Eight categories of PII detection: Identifies sensitive information such as private names, addresses, emails, phone numbers, URLs, dates, account numbers (including bank cards/credit cards), and keys/passwords.
• Context-aware redaction: Distinguishes public information from private information based on deep language understanding, avoiding false positives or missed detections caused by simple rules.
• Local high-throughput processing: Completes full-sequence labeling in a single forward pass, supporting uninterrupted processing of 128k long texts.
• Configurable precision/recall: Provides multiple operating point presets at runtime, allowing adjustment of redaction boundaries and strictness as needed.
• Supports fine-tuning: Can quickly adapt with a small amount of domain data to improve detection accuracy in specific scenarios (such as healthcare and finance).

Technical principles of OpenAI Privacy Filter

• Bidirectional token classification architecture: Uses an autoregressive pretrained model as the base, replaces the language modeling head with a token classification head, and performs post-training with a supervised classification loss.
• BIOES span decoding: The model outputs 33 token-level categories (8 privacy labels × 4 boundary labels + background class), and converts independent predictions into coherent span boundaries through a constrained Viterbi decoder.
• Banded attention mechanism: The model uses banded attention (bandwidth 128, effective window 257 tokens), balancing long context and computational efficiency.
• Sparse mixture of experts: 8-layer Transformer, each group query attention is equipped with 14 query heads and 2 KV heads, and the FFN layer uses a sparse MoE structure with 128 experts and top-4 routing.
• Constrained sequence decoding calibration: Controls background retention and span entry/continuation/closure through linear-chain transition scores and six transition bias parameters, achieving global path optimization.

How to use OpenAI Privacy Filter

• Environment preparation: Via pip install After installation, obtain opf CLI tool; on first run, if there is no local model, it will automatically pull from Hugging Face.
• Single-item redaction: Enter on the command line opf "待处理文本", supports --device cpu to switch to CPU operation, or --checkpoint to specify a custom model path.
• Batch file processing: Use opf to process files directly, or combine with pipeline commands such as cat file | grep pattern | opf to implement complex workflows.
• Interactive mode: Starts interactive mode when there is no input, outputting a structured JSON preview with ANSI colors.
• Model evaluation: Run opf eval dataset.jsonl to test precision and recall on an annotated dataset.
• Domain fine-tuning: Execute opf train train.jsonl --output-dir ./checkpoint You can fine-tune with your own data to adapt to enterprise-specific privacy policies.

Key Information and Usage Requirements of OpenAI Privacy Filter

• License: Apache 2.0, commercially usable, modifiable, and auditable.
• Runtime Environment: Supports GPU/CPU, can be deployed on laptops, browsers, and local servers.
• Language Limitations: Primarily trained on English; performance may degrade for non-Latin scripts, low-resource languages, and specific cultural naming patterns.
• Static Label Policy: Detection categories cannot be dynamically modified at runtime; label policies must be adjusted through fine-tuning.

Core Advantages of OpenAI Privacy Filter

• Privacy-First Local Deployment: Sensitive text in the model does not need to leave the device, reducing cloud leakage risks.
• Small Size, Big Capability: 1.5B/50M parameters achieve cutting-edge detection performance, suitable for edge devices.
• Native Long-Text Support: 128k context avoids boundary information loss caused by traditional chunking.
• Auditable and Customizable: Open-source weights and code support independent enterprise review, adapting to internal data governance requirements through fine-tuning.

Project Addresses of OpenAI Privacy Filter

• Project Official Website：https://openai.com/index/introducing-openai-privacy-filter/
• GitHub Repository：https://github.com/openai/privacy-filter
• HuggingFace Model Library：https://huggingface.co/openai/privacy-filter
• Technical Paper：https://cdn.openai.com/pdf/c66281ed-b638-456a-8ce1-97e9f5264a90/OpenAI-Privacy-Filter-Model-Card.pdf

Comparison of OpenAI Privacy Filter with Similar Competitors

Application Scenarios of OpenAI Privacy Filter

• AI Training Data Cleaning: Desensitize user conversations and documents before model training to prevent PII leakage into training sets.
• Log and Index Desensitization: Perform real-time or batch desensitization on system logs and search engine indexes to meet the principle of data minimization.
• Customer service and medical record processing: Automatically masks sensitive patient or customer information in ticket and medical record analysis workflows.
• Code repository secret scanning: Detects secrets such as API keys and passwords in code to prevent accidental commits to version control.
• Compliance pre-review assistance: Serves as an automated initial screening layer in compliance processes such as GDPR and CCPA, reducing the manual review burden.