zerox

Purpose

Zerox provides an extremely simple way to perform OCR recognition on documents so that AI can ingest them. Documents are essentially a visual presentation, and when faced with unusual layouts, tables, charts, and other content, visual models happen to be useful.

Main capabilities

• Accepts input files (PDF, DOCX, images, etc.).
• Converts files into a series of images.
• Passes each image to a visual model such as GPT to request Markdown generation.
• Aggregates all responses and returns Markdown.

Supported models

Zerox supports a range of models across different providers:

Azure OpenAI:

• GPT-4 Vision (gpt-4o)
• GPT-4 Vision Mini (gpt-4o-mini)
• GPT-4.1 (gpt-4.1)
• GPT-4.1 Mini (gpt-4.1-mini)

OpenAI:

• GPT-4 Vision (gpt-4o)
• GPT-4 Vision Mini (gpt-4o-mini)
• GPT-4.1 (gpt-4.1)
• GPT-4.1 Mini (gpt-4.1-mini)

AWS Bedrock:

• Claude 3 Haiku (2024.03, 2024.10)
• Claude 3 Sonnet (2024.02, 2024.06, 2024.10)
• Claude 3 Opus (2024.02)

Google Gemini:

• Gemini 1.5 (Flash, Flash-8B, Pro)
• Gemini 2.0 (Flash, Flash-Lite)

Usage

Zerox provides both Node and Python packages.

Node.js SDK

Supports visual models from different providers, such as OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, Google Gemini, etc. Zerox uses graphicsmagick and ghostscript in the PDF-to-image step; these are usually pulled automatically, but you may need to install them manually.

When used from a local path, the maintainFormat option attempts to return consistently formatted Markdown by passing the output of the previous page as additional context for the next page. This requires requests to run synchronously, so it is much slower, but if the document contains a lot of tabular data, or tables often span pages, this option is valuable.

Data extraction

Zerox supports extracting structured data from documents using a schema. This enables you to pull specific information from documents in a structured format, rather than only getting the full Markdown conversion result.

• Set extractOnly: true and provide a schema to extract structured data. The schema follows the JSON Schema standard.
• Use extractPerPage to extract data page by page, rather than extracting the entire document at once.
• You can also set extractionModel, extractionModelProvider, and extractionCredentials to use a different model for extraction than for OCR. By default, the same model is used.

Python SDK

Supports visual models from different providers, such as OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, etc.

Installation steps:

1. Install poppler on your system; it should be in the path variable. See the pdf2image documentation for instructions for each platform.
2. Install py-zerox:

The pyzerox.zerox function is an asynchronous API that uses a visual model to perform OCR-to-Markdown conversion. It processes PDF files and converts them to Markdown format. Before using this API, make sure to set the environment variables for the model and model provider. For environment setup and passing the correct model name, refer to the LiteLLM documentation.

Parameter description:

• cleanup (bool, optional): Whether to clean up temporary files after processing is complete. Default True.
• concurrency (int, optional): The number of concurrently running processes. Default 10.
• file_path (Optional[str], optional): The path of the PDF file to process. Defaults to an empty string.
• maintain_format (bool, optional): Whether to maintain formatting consistent with the previous page. Default False.
• model (str, optional): Used to generate completionsmodel. Default "gpt-4o-mini". Please refer to LiteLLM Providers to get the correct model name; the name may vary by provider.
• output_dir (Optional[str], optional): The directory to save the Markdown output. Default is None.
• temp_dir (str, optional): The directory for storing temporary files, defaulting to a named folder under the system temporary directory. If it already exists, Zerox will delete its contents before use.
• custom_system_prompt (str, optional): The system prompt used for the model, which overrides Zerox's default system prompt. Generally not needed unless some specific behavior is required. Default is None.
• select_pages (Optional[Union[int, Iterable[int]]], optional): The pages to process, which can be a single page number or an iterable of page numbers. Default is None.
• kwargs (dict, optional): Additional keyword arguments passed to the litellm.completion method. See the LiteLLM documentation and Completion Input for details.

Return value: ZeroxOutput, containing the Markdown content generated by the model and some metadata.

Example output comes from "azure/gpt-4o-mini". Note that for easier reading of the documentation, the output has been manually line-wrapped.

Supported file types

Zerox uses libreoffice and graphicsmagick together for document-to-image conversion. For non-image, non-PDF files, libreoffice is first used to convert the file to PDF, and then to an image.

Applicable scenarios and limitations

Applicable scenarios include: converting documents such as PDFs, DOCX, and images with complex layouts, tables, or charts into Markdown for subsequent AI processing; extracting specific information from documents in a structured way according to a schema; processing by page or limiting the page numbers to process; and scenarios with many cross-page tables where formatting consistency needs to be maintained.

Limitations and notes:

• The maintainFormat option requires requests to run synchronously, which will be much slower.
• Before use, the corresponding dependencies (such as graphicsmagick, ghostscript, poppler) and model environment need to be installed and configuredvariables.
• In the Python SDK, the default value of model is "gpt-4o-mini", but the actual model name may vary by provider, so refer to the LiteLLM documentation.
• Regarding pricing, charges, specific effects, and performance data, the materials do not currently provide them.

This project uses the MIT license.