SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsmarkitdown-ocr

markitdown-ocr

Office & productivity

View repository

Updated 10-15 21:48

About this skill

MarkItDown OCR (LLM Vision) Complete Usage Guide

Clear purpose: enable the LLM to, based on this skill, correctly configure and invoke markitdown-ocr plugin, to complete OCR extraction of embedded images and scanned pages in PDF/DOCX/PPTX/XLSX.

Offline baseline: based on packages/markitdown-ocr/ main branch (as of 2026-08). For any question like "does the plugin currently support X / has the parameter name changed", refer to the official README and source code:

When to Use This Skill

✅ When to use this skill(LLM Vision OCR / scanned documents / embedded images):

  • User asks "how do I convert a scanned PDF to markdown" (the original PDF has no selectable text)
  • User asks "how do I extract text from images embedded in PPT/Word/Excel"
  • User wants to use GPT-4o / GPT-4o-mini / Claude / Gemini for OCR
  • User asks "how do I configure markitdown-ocr with Azure OpenAI"
  • User asks "how do I install and use the markitdown-ocr plugin"
  • User wants to do local OCR (via vLLM / Ollama)

❌ When not to use this skill:

  • Only ordinary CLI usage (converting PDF / DOCX, with no image text needs) → markitdown-cli
  • User asks "what is MarkItDown / which formats does it support / how do I choose" → markitdown-awesome
  • User asks "how do I configure Azure Document Intelligence" → markitdown-cli(that is a built-in MarkItDown backend, not this OCR plugin)
  • User asks "how do I convert a PDF to plain text" (textract is more suitable) → hand off to markitdown-awesome(references/ecosystem/alternatives.md to give a comparison of markitdown / textract / unstructured / pandoc / docling)

⚠️ Gray areas:

  • User says "convert PDF to markdown but the image text is missing" → by default go the OCR plugin route (this skill)
  • User says "I want to use an LLM to extract tables" → OCR can do it, but if table accuracy requirements are high, prefer Azure Document Intelligence
  • User is already using another OCR (Tesseract / PaddleOCR) → outside this skill's scope, guide them to compare tools

Workflow

Every time you answer an OCR-type question, go through the following process once:

Step 1 — Confirm the document type & the user's goal

  • Input format: PDF / DOCX / PPTX / XLSX (this skill covers only these four)
  • Text location: embedded images / full scanned pages / both
  • LLM selection: OpenAI / Azure / Gemini / local
  • Cost budget: expensive vs cheap (gpt-4o vs gpt-4o-mini)

Step 2 — Check the environment

pip show markitdown-ocr openai | grep -E '^(Name|Version)'
markitdown --list-plugins | grep ocr

Both should produce output.

Step 3 — Choose the Python API form

  • OpenAI:MarkItDown(enable_plugins=True, llm_client=OpenAI(), llm_model="gpt-4o")
  • Azure OpenAI: switch AzureOpenAI(...), pass azure_endpoint / api_version
  • Local vLLM / Ollama:OpenAI(base_url=..., api_key=<your-token>)(local services usually do not validate the key)
  • Gemini:OpenAI(base_url=..., api_key=<your-key>)

Step 4 — Decide on the Prompt

  • Default prompt (bundled with the plugin): "Extract all text … maintain layout … no commentary"
  • To emphasize tables: add "Preserve table layout with Markdown pipes"
  • To emphasize outputting only text: add "Do not describe, summarize, or comment"

Step 5 — Self-check & run

Refer to the Validation self-check checklist。

Step 6 — Processing result

  • Output contains *[Image OCR]…[End OCR]* block → success
  • Not present → check the plugin list, client, and model name (see §8 Troubleshooting)

Validation (self-check list)

Run through this before/after answering any OCR-type question:

  • Has the user installed markitdown-ocr?(pip show markitdown-ocr)
  • Has the user installed the OpenAI-compatible SDK? (openai / azure-openai / custom)
  • Has the user configured OPENAI_API_KEY(or equivalent credentials)?
  • In the Python code,at the same timeprovided enable_plugins=True and llm_client and llm_model?(If any of the three is missing, OCR is skipped)
  • Does the model support vision? (You cannot use a text-only model such as gpt-3.5-turbo)
  • When using Azure OpenAI, azure_endpoint does not have a trailing /、api_version valid?
  • Handling sensitive documents → recommend a private deployment backend (vLLM / Ollama) instead of a cloud API?
  • Did you avoid falsely promising that the CLI can configure --llm-client/--llm-model?(These two CLIs do not exist; Python wrapping is required)

Gotchas

From highest to lowest hit frequency:

  1. The CLI does not have --llm-client / --llm-model — any claim of "enabling OCR directly from the command line" is wrong. Either write a Python wrapper or tell the user to use the Python API.__main__.py in the argparse registry.
  2. enable_plugins=True omitted — MarkItDown defaults to enable_plugins=False, and the plugin will not be registered at all.must be included。
  3. Missing llm_client or llm_model — the plugin "will still be registered" (without an OCR service), but the actual conversion will silently skip OCR. Checklist: all three parametersmustbe provided.
  4. LLM failures are silently swallowed — _ocr_service.py::extract_text converts exceptions into OCRResult(text="", error=str(e)); the plugin's convert() continues and does not throw. Symptom: the OCR block for a certain image is missing, but the rest of the document is OK.There is no stderr warning at all。
  5. PPTX descriptions take precedence over OCR — if the LLM returns an "image description", OCR is only a fallback. This means: for the same image, different calls may produce different text (description vs OCR).
  6. XLSX positional context is lost — images are uniformly appended after the data table (### Images in this sheet:), and are not interleaved with rows. Use with caution in table-heavy scenarios.
  7. Scanned PDFs render full pages at 300 DPI — much more expensive than "embedded small images"; a 50-page scan ≈ 50 vision calls, with high token cost.
  8. OCR blocks are identified by regex — the output format *[Image OCR]\n…\n[End OCR]* is a public contract; when post-processing depends on it, modifying the plugin source code will break the regex.
  9. Local vision models may not support data: URI — support for Base64-embedded images is unstable in some vLLM / Ollama models; if it fails, confirm that the model itself supports it.
  10. Azure OpenAI api_version mismatch — the deployment and SDK version must be compatible; using too new a version number may result in 404.
  11. One LLM call per image = high concurrency risk — using a thread pool directly for 100 images may hit OpenAI RPM/TPM limits. Swallowed 429s manifest as "images not OCR'd" and are hard to notice.
  12. OCR blocks pollute the main text summary — if you want to "see only the main text", you must first strip *[Image OCR]…[End OCR]*with regex, otherwise the LLM will cite OCR text as main text.
  13. gpt-3.5-turbo does not support vision — When the user enters the wrong model name, the API returns a 400, but the plugin silently swallows it; the symptom is that all OCR blocks are missing.
  14. Missing Pillow or PyMuPDF — markitdown-ocr is already in the dependency list; if you carelessly pip install markitdown-ocr --no-deps these will be missing, causing import to directly report an error.
  15. Single memory spike for large files — When the entire PDF goes through convert_local , PyMuPDF rendering + base64 encoding are in memory at the same time; images of tens of MB will blow up memory.

1. What it is / what problem it solves

MarkItDown's built-inPDF/DOCX/PPTX/XLSX convertersdo notperform OCR on embedded images or scanned pages—they only take "text that is already parseable". If the document contains images, scanned pages, or text converted into images, that part is empty in the default output.

markitdown-ocr plugintakes over the above four built-in converters (priority -1.0, higher than the built-in 0.0 ), and via LLM Vision(any OpenAI-compatible model that supports image input) extracts text from embedded images and inserts it back into the Markdown stream according to the format.

Core features:

  • ✅ Full support for PDF / DOCX / PPTX / XLSX
  • ✅ Automatic full-page OCR fallback for scanned PDFs (render each page at 300 DPI and send the whole page to the LLM)
  • ✅ DOCX image relationship graph fully preserved (paragraph/heading/table flow not disrupted)
  • ✅ PPTX processes images/placeholders/images within groups in "top→left" reading order
  • ✅ XLSX restores column/row letter coordinates by image anchor, and images are uniformly appended after the table
  • ✅ Output wrapped with *[Image OCR]…[End OCR]* around each OCR block, convenient for post-processing
  • ❌ Does not introduce new ML libraries or binary dependencies (the dependencies are already in the built-in converters)
  • ⚠️ When the LLM call failssilently skip, and conversion continues

2. Installation

pip install markitdown-ocr
# 同时确保有可用的 OpenAI 兼容客户端
pip install openai            # 官方 OpenAI
# 或
pip install azure-openai      # Azure OpenAI(若走 Azure)

Install from source (development/debugging):

git clone https://github.com/microsoft/markitdown.git
cd markitdown/packages/markitdown-ocr
pip install -e .

Dependencies (pulled automatically):markitdown>=0.1.0、pdfminer.six、pdfplumber、PyMuPDF、mammoth、python-docx、python-pptx、pandas、openpyxl、Pillow。

3. Configuration — LLM client and model

Key constraint: The plugin itselfdoes not read any API key—it gets the client and model from MarkItDown(llm_client=…, llm_model=…, llm_prompt=…) , and authentication is entirely handled by the client (via its own environment variables/credential chain).

3.1 OpenAI (default)

export OPENAI_API_KEY=sk-...

from openai import OpenAI
client = OpenAI()  # 自动读 OPENAI_API_KEY

3.2 Azure OpenAI

from openai import AzureOpenAI

client = AzureOpenAI(
    api_key=os.environ["AZURE_OPENAI_API_KEY"],    # 或读其它环境变量
    azure_endpoint="https://<res>.openai.azure.com/",
    api_version="2024-02-01",
)

3.3 Other OpenAI-compatible services (Gemini / Ollama / vLLM / LM Studio / DeepSeek-VL, etc.)

Any object that implements the client.chat.completions.create(model=…, messages=[…]) interface can be passed in,provided it supports vision input. Example:

import openai
client = openai.OpenAI(
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
    api_key=os.environ["GOOGLE_API_KEY"],
)
# 之后正常使用,llm_model="gemini-2.0-flash" 等

For a more complete compatible client matrix, see backends/compatible-clients.md; for practical scripts, see examples/backends/azure-and-local.md。

3.4 Behavior when llm_client is missing

_plugin.py::register_converters in:

if llm_client and llm_model:
    ocr_service = LLMVisionOCRService(client=…, model=…, default_prompt=…)

If llm_client or llm_model is missing, the pluginis still registered(what is registered is the version without an OCR service), but during actual conversion itsilently skips OCR, falling back to the behavior of the built-in converter. Verify that the plugin exists:

markitdown --list-plugins
# 应当看到: ocr   (package: markitdown_ocr)

4. Python API — three typical usages

4.1 Simplest (OpenAI + gpt-4o)

from markitdown import MarkItDown
from openai import OpenAI

md = MarkItDown(
    enable_plugins=True,        # ← 必须
    llm_client=OpenAI(),        # ← 必须
    llm_model="gpt-4o",         # ← 必须
)
result = md.convert("document_with_images.pdf")
print(result.markdown)

4.2 Azure OpenAI

from markitdown import MarkItDown
from openai import AzureOpenAI

md = MarkItDown(
    enable_plugins=True,
    llm_client=AzureOpenAI(
        api_key=os.environ["AZURE_OPENAI_API_KEY"],
        azure_endpoint="https://<res>.openai.azure.com/",
        api_version="2024-02-01",
    ),
    llm_model="gpt-4o",
)
print(md.convert("report.docx").markdown)

4.3 Custom Prompt

md = MarkItDown(
    enable_plugins=True,
    llm_client=OpenAI(),
    llm_model="gpt-4o",
    llm_prompt="Extract all text from this image, preserving table structure.",
)

5. CLI — currently there is no native --llm-client / --llm-model

⚠️ Important:markitdown The current CLI (__main__.py) is not registered --llm-client / --llm-model. To enable OCR via the command line,write a 3-line wrapper in PythonThat is sufficient.

5.1 Recommended practice: shell wrapper

Write a small file mdocr:

#!/usr/bin/env python3
# mdocr — 命令行包装 markitdown-ocr
import os, sys, argparse
from markitdown import MarkItDown

ap = argparse.ArgumentParser()
ap.add_argument("filename")
ap.add_argument("-o", "--output")
ap.add_argument("--model", default=os.environ.get("MD_OCR_MODEL", "gpt-4o"))
ap.add_argument("--prompt", default=None)
args = ap.parse_args()

client = OpenAI()  # ← 替换为你要用的客户端构造
md = MarkItDown(
    enable_plugins=True,
    llm_client=client,
    llm_model=args.model,
    llm_prompt=args.prompt,
)
# 注意:请在环境变量 OPENAI_API_KEY / AZURE_OPENAI_API_KEY / OLLAMA_HOST 等中配置凭据
# 切勿把真实 key 写进源代码或 commit。
result = md.convert(args.filename)
sys.stdout.write(result.markdown)
if args.output:
    with open(args.output, "w", encoding="utf-8") as f:
        f.write(result.markdown)

chmod +x mdocr
./mdocr scan.pdf -o scan.md --model gpt-4o

5.2 One-liner python -c(for temporary use)

python -c "
from markitdown import MarkItDown
from openai import OpenAI
import sys
md = MarkItDown(enable_plugins=True, llm_client=OpenAI(), llm_model='gpt-4o')
sys.stdout.write(md.convert(sys.argv[1]).markdown)
" scan.pdf > scan.md

6. Format-level behavior differences (how OCR results are inserted back)

Format OCR block insertion location Description
PDF Inline (interspersed in the surrounding text), inserted in "top-to-bottom vertical reading order"; scanned pages are sent to the LLM as full pages Embedded images are extracted by page.images / page XObjects; for malformed PDF, automatically switches to full-page rendering with PyMuPDF
DOCX OCR is performed in the "DOCX→HTML→Markdown" pipelinebeforeexecution; the document flow (headings/paragraphs/tables) is 100% preserved Via doc.part.rels extract image relationships
PPTX In "top→left" reading order within the slide; if the LLM returns an "image description", the description is preferred, and OCR is used only as a fallback Supports Picture shape, placeholder shape, and images within groups
XLSX The images of each worksheet are uniformly appended after the data table and labeled as ### Images in this sheet: The position is calculated from the image anchor to column letters/row numbers;the positional context will be lost(different from the other three formats)

OCR output format: each block is wrapped in Markdown emphasis for easy identification during post-processing:

*[Image OCR]
<抽取到的文字>
[End OCR]*

To batch strip OCR blocks and create a "body text only" summary, you can use a one-line sed:

sed -e '/^\*\[Image OCR\]$/,/^\[End OCR\]\*$/d' input.md > cleaned.md

7. Limitations and costs (must-read before deployment)

Item Description
One LLM call per image 100 images = 100 vision calls; cost grows linearly with the number of images
Scanned PDFs are more expensive Scanned pages are sent to the LLM as full pages at 300 DPI, consuming far more tokens than "small embedded images"
LLM errors are silent "If the LLM call fails, conversion continues without that image's text"—failures are swallowed and require monitoring
PPTX descriptions take priority If the LLM returns an "image description", OCR is used only as a fallback, and the final text is the description rather than the original text
XLSX position is lost Images are appended after the table and will not appear near the cell where they were originally located
Image MIME The OCR service uses stream_info.mimetype or PIL detection; if neither recognizes it, it is treated as image/png

8. Troubleshooting quick reference

Symptom Troubleshooting
Not a single OCR block appears markitdown --list-plugins Check whether it contains ocr; confirm that enable_plugins=True and llm_client + llm_model
Reports ModuleNotFoundError: markitdown_ocr pip install markitdown-ocr, or pip install -e packages/markitdown-ocr
Reports ModuleNotFoundError: openai pip install openai or switch to another OpenAI-compatible SDK
OCR is missing some images That LLM call failed and was silently swallowed; add logging/retries, or switch to a more stable model
BadRequestError: image_url invalid The client/MIME is not recognized; in _ocr_service.py confirm that content_type is inferred correctly, or switch clients
Azure OpenAI authentication fails Confirm that api_version is a version supported by that deployment (e.g., 2024-02-01)、azure_endpoint without a trailing /
All data:image entries in the output are gone is the behavior of the MarkItDown main body and is unrelated to OCR — add --keep-data-uris(CLI) or in the Python API pass keep_data_uris=True

9. Testing / Secondary Development

# 跑插件自带的测试套件
cd packages/markitdown-ocr
pytest tests/ -v

# 本地装好后做最小冒烟
python -c "
from markitdown import MarkItDown
from openai import OpenAI
print(MarkItDown(enable_plugins=True, llm_client=OpenAI(), llm_model='gpt-4o').convert('sample.pdf').markdown[:500])
"

For the plugin's internal mechanisms (registration process, OCRService implementation details, differences between converters), see plugin/internals.md。

10. When to Switch to a Sibling Skill

User Question Switch to
"What is MarkItDown? Which formats does it support?" markitdown-awesome
"How do I use the markitdown command and pass parameters?" markitdown-cli
"I use Azure Document Intelligence / Content Understanding and don't need the OCR plugin" markitdown-cli 's "Azure Backend" section

11. Privacy Policy

This Skill provides local documents and sample code and does not collect, store, or transmit any user data.Actual OCR calls will send the image content you pass in (which may contain sensitive information) to the selected LLM service provider; please comply with the corresponding service provider's privacy terms and the regulations of the user's region. For production deployment, it is recommended to:

  • Use a privatized LLM for highly sensitive documents (such as local vLLM / Ollama + OpenAI-compatible protocol)
  • Disable request logging / temporary auditing

12. Reference Links