SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsminicpmvocr-sdk

minicpmvocr-sdk

Data & analytics

View repository

Updated 10-28 07:00

About this skill

MiniCPM-OCR 1.0 Deployment

SDK（minicpmvocr) handles layout detection (PP-DocLayoutV3), chunking, stitching, and concurrent scheduling, and calls the inference service over HTTP. The SDK and the inference service are installed in separate venvs.

1. Determine the approach

First ask the user which inference backend to use and whether the SDK and the service are on the same machine. When the user has no preference, use the default approach.

User goal Environment
Parse documents (default) sdk-env + vllm-ocr-env, DSpark-accelerated version
Do not want to compile vLLM sdk-env + tf-ocr-env
Already have a llama.cpp / OpenAI-compatible service Install only sdk-env
Verbatim verification on both vLLM and transformers Two inference environments, or a merged ocr-env(see Section 7)
  • Install only one inference environment, unless the user wants to verify on both backends.
  • sdk-env cannot be installed together with torch / vLLM, otherwise the numpy and opencv versions will conflict.
  • The SDK can run on a machine without a GPU; use --server-url or an SSH tunnel to connect to the remote service.

2. Pre-installation checks

nvidia-smi                                                      # 显存 ≥ 24 GB，驱动 ≥ 580
nvidia-smi --query-compute-apps=pid,used_memory --format=csv    # 找一张空闲卡
df -h .                                                         # 剩余 ≥ 20 GB
  • Drivers 535–579 cannot run the default combination (torch cu130). Tell the user they need to upgrade the driver, or switch to torch cu126 and compile vLLM themselves (verified on A100; not covered by this skill).
  • The vLLM environment requires Python ≥ 3.11; the SDK and transformers environments require ≥ 3.10.

Paths that need to be provided by the user (ask first if not given; do not guess):

Placeholder Meaning
<SDK_REPO> Official repository URL:https://github.com/OpenBMB/MiniCPM-OCR.git
<VLLM_SRC> feat-ocr vLLM source directory (commit 5c2b92a5dc）
<TRANSFORMERS_SRC> feat-ocr transformers source directory (commit 69eafbc5bd）

The following commands are all run in the same working directory: environments are created under ./sdk-env and other directories, models under ./models/, the repository is cloned to ./minicpmvocr-sdk/, and the SDK source is in its sdk/ subdirectory.

3. Installation

3.1 sdk-env

python3 -m venv sdk-env && source sdk-env/bin/activate
pip install paddlepaddle-gpu==3.3.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu129/
# 没有 GPU 时改装 CPU 版：pip install paddlepaddle==3.2.1
git clone <SDK_REPO> minicpmvocr-sdk
pip install -e "./minicpmvocr-sdk/sdk[all]"
pip install paddlex==3.4.2
minicpmvocr list-backends

Pass criteria: the output contains the vllm、llamacpp、openai backend and the ppdoclayout3_paddlex、ppdoclayout3_torch、ppdoclayout3_server three layout providers.

If the CPU version of paddle is installed, then every subsequent minicpmvocr command must add --set layout.device=cpu。

3.2 vllm-ocr-env

sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt-get update && sudo apt-get install -y gcc-13 g++-13 libdw-dev libelf-dev

python3.13 -m venv vllm-ocr-env && source vllm-ocr-env/bin/activate
pip install torch==2.14.0 torchaudio==2.11.0 torchvision==0.29.0
pip install "transformers>=5.16.1,<5.20.0"
pip install "setuptools-rust>=1.9.0" "setuptools-scm>=8.0" "setuptools>=77.0.3,<81.0.0" \
    "cmake>=3.26.1" ninja packaging wheel jinja2
pip install nvidia-cuda-nvcc==13.0.48 nvidia-cuda-crt==13.0.48 nvidia-nvvm==13.0.48 \
    nvidia-cuda-runtime==13.0.96 nvidia-cuda-nvrtc==13.0.88 nvidia-nvjitlink==13.4.92 \
    nvidia-cuda-cccl==13.3.4.3.1

nvidia-cuda-cccl cannot be omitted; if missing, it reports cuda_fp16.h: nv/target: No such file or directory。

Complete the directory layout for pip CUDA packages:

CU="$VIRTUAL_ENV/lib/python$(python -c 'import sys; print("%d.%d" % sys.version_info[:2])')/site-packages/nvidia/cu13"
mkdir -p "$CU/nvvm/bin" && ln -sf "$CU/bin/cicc" "$CU/nvvm/bin/cicc"
ln -sfn lib "$CU/lib64"
ln -sf "$CU/lib/libcudart.so.13" "$CU/lib/libcudart.so"
ln -sf "$CU/lib/libnvrtc.so.13" "$CU/lib/libnvrtc.so"
mkdir -p "$CU/lib64/stubs" && ln -sf /usr/lib/x86_64-linux-gnu/libcuda.so.1 "$CU/lib64/stubs/libcuda.so"

If the 13.0.48 package is missing bin/crt/link.stub、prelink.stub or bin/nvdisasm, copy them from an environment on the same machine that has 13.4.92 installed to the corresponding location under $CU/bin/ .

Compilation (about 20 minutes on 96 cores; lower it when memory is small: MAX_JOBS）：

cd <VLLM_SRC>
# 无法访问 GitHub 时，先设置下面“离线编译”中的变量
CUDA_HOME="$CU" PATH="$VIRTUAL_ENV/bin:$CU/bin:$PATH" \
  CC=gcc-13 CXX=g++-13 CUDAHOSTCXX=g++-13 MAX_JOBS=64 \
  pip install -e . --no-build-isolation --progress-bar off
cd -
python -c "from vllm import LLM; print('vLLM OK')"

Do not use import vllm._C to determine whether compilation succeeded. This branch's extension name is _C_stable_libtorch.abi3.so。

The DSpark main model uses the minicpm_ocr architecture, which the transformers on PyPI does not support. Install feat-ocr transformers after compiling vLLM; the order cannot be reversed:

pip install --no-deps -e <TRANSFORMERS_SRC>
python -c "import transformers; print(transformers.__version__)"   # 应为 5.19.0.dev0

Offline compilation (when CMake times out fetching dependencies such as CUTLASS):

DEPS=<已有 vLLM 源码>/.deps
export VLLM_CUTLASS_SRC_DIR="$DEPS/cutlass-src"
export VLLM_FLASH_ATTN_SRC_DIR="$DEPS/vllm-flash-attn-src"
export FLASH_KDA_SRC_DIR="$DEPS/flashkda-src"
export DEEPSELECT_SRC_DIR="$DEPS/deepselect-src"
export FLASH_MLA_SRC_DIR="$DEPS/flashmla-src"
export TRITON_KERNELS_SRC_DIR="$DEPS/triton_kernels-src/python/triton_kernels/triton_kernels"
export TML_FA4_SRC_DIR="$DEPS/tml_fa4-src"
export FMHA_SM100_SRC_DIR="$DEPS/fmha_sm100-src"
export QUTLASS_SRC_DIR="$DEPS/qutlass-src"
export DEEPGEMM_SRC_DIR="$DEPS/deepgemm-src"
export CMAKE_ARGS="-DCUTLASS_ENABLE_TESTS=OFF"   # 否则还会拉取 googletest

Warnings that can be ignored after compilation:

  • vllm.third_party.deep_gemm ... undefined symbol: _ZSt8to_charsPcS_DF128_: the OCR model does not use DeepGEMM.
  • FlashInfer's precompiled kernels are missing or do not match: cold start will be slower (measured initialization is about 104 seconds), and results are not affected.

3.3 tf-ocr-env

python3 -m venv tf-ocr-env && source tf-ocr-env/bin/activate
pip install torch==2.14.0 torchvision
pip install -e <TRANSFORMERS_SRC>         # 需要 OpenAI 兼容服务时装 "<TRANSFORMERS_SRC>[serving]"
pip install accelerate
# 可选加速内核，不装时回退参考实现，结果相同但更慢：
# pip install causal-conv1d flash-linear-attention

3.4 Download the model

Main model:MiniCPM-OCR-1.0; DSpark draft model:MiniCPM-OCR-1.0-DSpark。

# DSpark：主模型和 draft 必须来自同一次发布，不能和其他 checkpoint 混用
HF_ENDPOINT=https://hf-mirror.com hf download openbmb/MiniCPM-OCR-1.0 --local-dir ./models/MiniCPM-OCR-1.0
HF_ENDPOINT=https://hf-mirror.com hf download openbmb/MiniCPM-OCR-1.0-DSpark --local-dir ./models/MiniCPM-OCR-1.0-DSpark
sha256sum ./models/MiniCPM-OCR-1.0-DSpark/model.safetensors   # 对照 release_manifest.json 的 weight_sha256

curl -L -o PP-DocLayoutV3_infer.tar \
  https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-DocLayoutV3_infer.tar
tar xf PP-DocLayoutV3_infer.tar
export OCR_LAYOUT_MODEL_DIR=$PWD/PP-DocLayoutV3_infer
test -f "$OCR_LAYOUT_MODEL_DIR/inference.pdiparams" && echo "layout OK"
  • ModelScope：modelscope download --model <MS_REPO_ID> --local_dir <同名目录>。
  • The main model directory should contain config.json、model.safetensors、tokenizer.json、preprocessor_config.json、chat_template.jinja。
  • OCR_LAYOUT_MODEL_DIR must be a Paddle inference directory (containing inference.pdiparams、inference.json、inference.yml), not a safetensors directory.

4. Start the inference service

Before starting vLLM in each new shell, set CUDA_HOME, otherwise the first startup on Blackwell GPUs will report Could not find nvcc：

source vllm-ocr-env/bin/activate
CU="$VIRTUAL_ENV/lib/python$(python -c 'import sys; print("%d.%d" % sys.version_info[:2])')/site-packages/nvidia/cu13"
export CUDA_HOME="$CU" PATH="$VIRTUAL_ENV/bin:$CU/bin:$PATH"

4.1 DSpark accelerated version (default)

CUDA_VISIBLE_DEVICES=<空闲卡> python -m vllm.entrypoints.openai.api_server \
  --model ./models/MiniCPM-OCR-1.0 \
  --served-model-name minicpm-ocr \
  --port 18499 \
  --max-model-len 40960 \
  --max-num-batched-tokens 40960 \
  --max-num-seqs 1 \
  --attention-backend FLASH_ATTN \
  --mm-encoder-attn-backend FLASH_ATTN \
  --mm-processor-kwargs '{"downsample_mode":"4x"}' \
  --gpu-memory-utilization 0.85 \
  --trust-request-mm-kwargs \
  --speculative-config '{"method":"dspark","model":"./models/MiniCPM-OCR-1.0-DSpark","num_speculative_tokens":15,"enable_adaptive_verification":false}' \
  --default-chat-template-kwargs '{"enable_thinking":false}'
  • --max-num-seqs 1 means only one request is processed at a time; concurrent requests from the SDK will queue up. When the user needs high throughput, start multiple instances and repeatedly pass --server-url, or switch to the standard version.
  • When starting the service, use --served-model-name minicpm-ocr; the SDK parses the command and correspondingly uses --model minicpm-ocr. When changing the name, both places must be changed together.

4.2 Standard version (no speculative sampling)

CUDA_VISIBLE_DEVICES=<空闲卡> python -m vllm.entrypoints.openai.api_server \
  --model ./models/MiniCPM-OCR-1.0 \
  --served-model-name minicpm-ocr \
  --port 18499 \
  --max-model-len 40960 \
  --limit-mm-per-prompt '{"image": 36}' \
  --gpu-memory-utilization 0.85 \
  --trust-request-mm-kwargs

Both versions must include --trust-request-mm-kwargs. If it is omitted, the service returns 400 for every request, and the SDK silently retries until it reports an error after 600 seconds, which looks like it is stuck.

4.3 transformers serve

source tf-ocr-env/bin/activate
transformers serve ./models/MiniCPM-OCR-1.0 --port 18499 --host 0.0.0.0 --continuous-batching

4.4 Readiness check

curl --noproxy 127.0.0.1 http://127.0.0.1:18499/v1/models     # 首次启动约 3 分钟

Only when the response contains "id":"minicpm-ocr" is it considered ready. If the first startup times out during the CUDA graph capture stage or is terminated by SIGTERM, restart it once directly (the compilation cache has already been generated); if it still fails, add --enforce-eager。

5. Parse documents

source sdk-env/bin/activate
export OCR_LAYOUT_MODEL_DIR=$PWD/PP-DocLayoutV3_infer
export PADDLE_PDX_DISABLE_MODEL_SOURCE_CHECK=True

minicpmvocr parse --config minicpmvocr-sdk/configs/backends/vllm.yaml \
  --server-url http://127.0.0.1:18499 \
  --model minicpm-ocr \
  --set input.limit=1 --input <文件> --output out.md
  • First use --set input.limit=1 to parse only one page; after confirming it passes, remove it and parse all pages.
  • The configuration files are all in minicpmvocr-sdk/configs/backends/: for vLLM use vllm.yaml; for transformers serve and other OpenAI-compatible services use openai.yaml; for llama.cpp use llamacpp.yaml。
  • When using openai.yaml with --set engine.extra_body='{"chat_template_kwargs":{"enable_thinking":false}}', otherwise the output may be empty or garbled.vllm.yaml has thinking mode automatically disabled.
  • On multi-GPU machines, layout detection uses GPU 0 by default. When GPU 0 is occupied, add --set layout.device=cuda:N。
  • Except for parse , all commands (run、parse-layout、prepare、infer、show-config) must use --set input.file_path、input.files_dir or input.image_dir to specify the input.
  • --set are resolved relative to the directory containing the configuration file; always write absolute paths (e.g. $PWD/doc.pdf）。--input is resolved relative to the current directory.
  • run The output directory of run.run_tag in the configuration; pages that already have results are skipped. Before rerunning, change --set run.run_tag=<新名字>。
  • Artifacts are written by default under runs/ in the SDK directory. When disk space is tight, add --set run.run_root=<其他盘的绝对路径>。

6. Verify results

python - <<'PY'
from pathlib import Path
md = Path("out.md").read_text(encoding="utf-8")
assert md.strip(), "输出为空"
assert "<otsl>" not in md, "表格仍是 OTSL，没有转成 HTML"
print("OK", len(md), "chars", "| table" if "<table>" in md else "")
PY
  • Table pages should have <table>; formula pages should have LaTeX (e.g. \frac); body-text pages should be readable text.
  • For the DSpark version, also confirm that speculative sampling is taking effect. After sending a few requests:
curl --noproxy 127.0.0.1 -s http://127.0.0.1:18499/metrics | grep spec_decode_num

You should see spec_decode_num_accepted_tokens_total and spec_decode_num_drafts_total, and the values should increase with requests. accept length = 1 + accepted / drafts; in the validation environment it is 5.2–7.0. If these two items are absent, the service was not started with --speculative-config. The service responding does not mean DSpark is taking effect.

7. Optional: word-by-word verification on both ends

Do this only when the user requests it.verify_transformers.py and verify_vllm.py are provided with the feat-ocr source code and are not in the SDK. If the user does not have these two scripts, skip it and do not claim that both ends have been verified to be consistent.

If you want to run both ends in one environment, execute in the environment where vLLM has been compiled (the order cannot be reversed):

pip install --no-deps -e <TRANSFORMERS_SRC>
pip install --no-deps accelerate
python -c "import transformers, vllm, torch; print(transformers.__version__, vllm.__version__, torch.__version__)"

Run the two ends separately:

# transformers 端（tf-ocr-env 或合并环境）
CUDA_VISIBLE_DEVICES=<空闲卡> python verify_transformers.py \
  --model ./models/MiniCPM-OCR-1.0 --image table.png \
  --prompt "请识别图中的表格并输出 OTSL" --attn sdpa --max-new-tokens 512

# vLLM 端（vllm-ocr-env 或合并环境，先按第 4 节设置 CUDA_HOME）
CUDA_VISIBLE_DEVICES=<空闲卡> python verify_vllm.py \
  --model ./models/MiniCPM-OCR-1.0 --image table.png \
  --prompt "请识别图中的表格并输出 OTSL" --max-tokens 512

Pass criteria:

  • When loading on the transformers side, there are no missing, unexpected, or mismatched keys, and the parameter count is 1.300B.
  • Both ends have the same number of prompt tokens, and the outputs are identical character by character, both ending with </otsl> .

Reference values (1080×1080 table image): prompt 1975 tokens, vision 1936 tokens, generation 284 tokens, output 677 characters. The values change after switching images or prompts; the criterion is that both ends match.

8. Error handling

Installation and compilation

Symptom Handling
Failed to clone https://github.com/nvidia/cutlass.git Cannot access GitHub; use the offline compilation setup in 3.2 for *_SRC_DIR
After disabling CUTLASS fetching, still need to fetch googletest Set CMAKE_ARGS="-DCUTLASS_ENABLE_TESTS=OFF"
Could not find a package configuration file provided by "MLIR" TRITON_KERNELS_SRC_DIR level is wrong; it should point to .../python/triton_kernels/triton_kernels
During compilation, many FAILED: [code=9] compile processes are killed due to insufficient memory. Lower MAX_JOBS(use 8–32 for 147 GB of memory), install first ccache to avoid recompiling
ld: cannot find -lcudart Add the lib64、libcudart.so、stubs/libcuda.so link from 3.2
CUDA_nvrtc_LIBRARY ... NOTFOUND Add libnvrtc.so link
cuda_fp16.h ... nv/target: No such file Install nvidia-cuda-cccl==13.3.4.3.1
No module named vllm._C It is not a compilation failure; verify with from vllm import LLM Verify
In the vLLM environment, transformers is 5.19.0 rather than 5.19.0.dev0 The editable install was overwritten by a vLLM dependency; reinstall pip install --no-deps -e <TRANSFORMERS_SRC>
Layout reports missing pypdfium2 pip install "pypdfium2>=4", which is also needed when only processing images

Start the service

Symptom Handling
model type 'minicpm_ocr' ... does not recognize this architecture The vLLM environment is missing feat-ocr transformers; install it per 3.2. No need to recompile vLLM
Could not find nvcc Not set CUDA_HOME, see Section 4
ptxas fatal: Unsupported .version 9.4 or CUDA compiler and CUDA toolkit headers are incompatible Unify nvcc, runtime, nvrtc, and nvjitlink to 13.4.92, then add the links per 3.2
flashinfer-cubin version does not match pip uninstall flashinfer-cubin；vllm download-kernels run it when it exists
CUDA driver version is insufficient Driver is lower than 580, see Section 2
When the driver is < 580, it reports TopKMaskLogits ... device kernel image is invalid Set VLLM_USE_FLASHINFER_SAMPLER=0; greedy decoding gives the same result
When the driver is < 580, it reports nvcc fatal: Unknown option '--compress-mode=size' CUDA_HOME point to a toolchain that matches torch (cu126 corresponds to 12.8)
When importing vLLM, it reports libtorchcodec_image.so pip uninstall torchcodec
Free memory on device ... less than desired GPU memory utilization This card is occupied; switch to CUDA_VISIBLE_DEVICES
OOM --gpu-memory-utilization lower to 0.7, or --max-model-len lower to 16384
CUDA graph capture times out or is terminated by SIGTERM Restart once; if it still fails, add --enforce-eager

Parsing

Symptom Handling
Parsing produces no result for a long time, and the log shows request failed until timeout The service omitted --trust-request-mm-kwargs; after the request was rejected, the SDK kept retrying until the 600-second timeout
After waiting for a long time, the SDK reports no ... endpoint became ready within 1800s The service is not up or the address is wrong; first check with curl /v1/models Check
Image requests return 400 The standard version omitted --limit-mm-per-prompt '{"image": 36}'
Output is empty or garbled Thinking mode is not disabled, see Section 5
The table is <otsl> instead of <table> It is not converted when calling the API directly; use the SDK pipeline
crop of a small image reports assign N multimodal tokens to M placeholders The vLLM version is too old; switch to the feat-ocr build
Layout reports a CUDA error The CPU version of paddle is installed; add --set layout.device=cpu
At least one of input.image_dir / input.files_dir / input.file_path is required Add --set input.file_path=<文件的绝对路径>
input.file_path not found, but the file exists --set uses a relative path; change it to an absolute path
run Reran it and the result is unchanged Switch to --set run.run_tag=<新名字>
pip timed out Switch to a domestic mirror. When accessing localhost, set no_proxy=127.0.0.1,localhost,::1
The service is on a remote machine ssh -L 18499:127.0.0.1:18499 user@host; for the SDK use --server-url http://127.0.0.1:18499

9. Verified versions

Component Version
vLLM environment Python 3.13.9, torch 2.14.0+cu130, vLLM source 5c2b92a5dc，feat-ocr transformers 69eafbc5bd
transformers environment Python 3.10.12 / 3.13.9，torch 2.14.0+cu130，transformers 5.19.0.dev0（69eafbc5bd）
SDK environment minicpmvocr 0.2.3, paddlepaddle-gpu 3.3.0 or paddlepaddle 3.2.1, paddlex 3.4.2
Hardware 3×RTX 5090, driver 590.48.01, Ubuntu 22.04.5 (2026-10-19)

10. Reporting rules

  • The 5090 is only the machine used for verification; do not say that a 5090 is required. The threshold is VRAM ≥ 24 GB and driver ≥ 580.
  • Report only the checks that were actually run. If you have not run verification on both ends, do not say the two ends are consistent; if you have not checked /metrics , do not say DSpark is in effect.
  • There is no Web UI script in the SDK repository; do not mention to users or guide users to run any demo_*.py。