multimodal-embedding-serving-user

Multimodal Embedding Serving — User

Run and call the embedding service. Run commands yourself and relay output. REST base URL: http://localhost:9777 (host port hardcoded by setup.sh; container 8000).

When to Use

• Deploy the embedding service (Docker Compose or standalone) and confirm health
• Embed text, images, or videos over REST on port 9777
• Choose or switch among the CLIP/SigLIP/MobileCLIP/CN-CLIP/Blip2/QwenText models
• Integrate embeddings in-process via the Python SDK wheel
• Diagnose 400/422 errors or model-capability mismatches

Example Prompts

Sample Problem-solving scenarios this skill handles end-to-end:

Example	Problem it solves
image-similarity-finder.md	Find visually/semantically similar images in a local folder
text-to-image-search.md	Search an image folder with natural-language queries ("Google Lens" for local media)
image-duplicate-detector.md	Flag duplicate / near-duplicate images in a folder for cleanup, QC, and dataset deduplication

Docs & deploy files — with or without a clone

All paths below are relative to microservices/multimodal-embedding-serving/ in the edge-ai-libraries repo. No clone? Fetch any of them from GitHub raw:

https://raw.githubusercontent.com/open-edge-platform/edge-ai-libraries/main/microservices/multimodal-embedding-serving/<path>

Load these existing docs only when needed:

Resource	Load when…
docs/user-guide/api-reference.md + docs/user-guide/api-docs/openapi.yaml	building non-text payloads (image/video, base64, segment_config) or parsing responses/errors
docs/user-guide/supported-models.md	choosing or switching models (dimensions, modalities, language, size)
docs/user-guide/sdk-usage.md + docs/user-guide/wheel-installation.md	integrating in-process via the Python SDK wheel
docs/user-guide/get-started.md	more curl examples and env-var tables
setup.sh, docker/compose.yaml	the deploy artifacts used below

1. Context routing — repo clone or standalone? REST or SDK?

• In-process Python integration wanted (no separate server): the service doubles as an SDK — build the wheel with poetry build (needs the repo) and use get_model_handler(...) + EmbeddingModel; see docs/user-guide/sdk-usage.md. Rule of thumb: default to REST; pick the SDK when a Python process embeds heavily and an HTTP hop per item would dominate. Note: MobileCLIP/Blip2 extras exist only in the Docker image, and the wheel is not on PyPI.
• Otherwise detect a clone:

  [ -f setup.sh ] && grep -q 'name = "multimodal-embedding-serving"' pyproject.toml 2>/dev/null \
    && echo REPO || echo STANDALONE
  

  REPO → Step 2 from the microservice root. STANDALONE → fetch the two deploy files, then the exact same Step 2:

  RAW=https://raw.githubusercontent.com/open-edge-platform/edge-ai-libraries/main/microservices/multimodal-embedding-serving
  mkdir -p embedding-serving/docker && cd embedding-serving
  curl -fsSL $RAW/setup.sh -o setup.sh
  curl -fsSL $RAW/docker/compose.yaml -o docker/compose.yaml
  

  Already running (curl -sf localhost:9777/health) → Step 3.

2. Bring-up (identical in both contexts)

1. Pick a model — default CLIP/clip-vit-b-32 (512-dim, text+image+video). Trade-offs: docs/user-guide/supported-models.md.
2. setup.sh must be sourced and hard-fails without EMBEDDING_MODEL_NAME. REGISTRY_URL=intel selects the prebuilt image; --no-build prevents a source build. Run in the background — first start downloads the model:

   bash -c 'export EMBEDDING_MODEL_NAME="CLIP/clip-vit-b-32" REGISTRY_URL=intel TAG=latest \
     && source setup.sh && docker compose -f docker/compose.yaml up -d --no-build'
   

   Intel GPU: also export EMBEDDING_DEVICE=GPU (setup.sh then auto-enables OpenVINO + THROUGHPUT mode).
3. Wait for readiness:

   until curl -sf http://localhost:9777/health; do sleep 5; done
   

3. Capability check (mandatory before non-text inputs)

curl -s http://localhost:9777/model/capabilities

QwenText models are text-only — image/video requests return 400. GET /model/current shows the exact loaded model id to use in requests.

4. Embed

Text (single string or list of strings):

curl -s http://localhost:9777/embeddings -H 'Content-Type: application/json' -d '{
  "model": "CLIP/clip-vit-b-32",
  "input": {"type": "text", "text": "a red truck at a loading dock"},
  "encoding_format": "float"
}'

Response: {"embedding": [...]} — a flat vector for text/image; a list of per-frame vectors for video inputs.

• model must equal the loaded model (else 400).
• Image: {"type":"image_url","image_url":"https://…"} (plain string, not a nested object) or image_base64. Video: video_url/video_base64/ video_frames with segment_config (num_frames default 64, extraction_fps, frame_indexes) — full shapes and examples: docs/user-guide/api-reference.md.

5. Stop / clean

• docker compose -f docker/compose.yaml down
• Volumes ov-models (model caches) and data-prep persist; removing them forces re-downloads — confirm with the user first.

Troubleshooting

Symptom	Likely cause → action
source setup.sh prints ERROR and stops	EMBEDDING_MODEL_NAME not exported → export it first
No response on 9777	still starting/downloading → docker logs -f multimodal-embedding-serving
400 "model mismatch"	request model ≠ loaded model → GET /model/current
400 unsupported modality on image/video	text-only model (QwenText) → switch model or send text
First non-text request slow	lazy OpenVINO conversion/compile → expected once
422 on /embeddings	malformed input union → check shapes in docs/user-guide/api-reference.md
Port 9777 busy	stop the conflicting service (EMBEDDING_SERVER_PORT is hardcoded by setup.sh)