byted-bytehouse-multimodal-search

ByteHouse Multimodal Retrieval Skill

Description

This Skill is based on ByteHouse's vector retrieval capabilities and Volcengine Ark multimodal vectorization models (default doubao-embedding-vision), providing fromvectorization → table creation → writing → retrievala complete reference implementation of multimodal retrieval. Core capabilities:

Multimodal vectorization: text / images / videos → unified-dimension vectors.
Vector table creation: supports indexes such as HNSW, IVF_FLAT, IVF_PQ, with optional full-text inverted index enabled.
Document writing: single / batch insert, with content_type、title、metadata。
Retrieval capabilities: vector retrieval, hybrid retrieval (vector + keywords), text-to-image search, image-to-image / video search.

Use this Skill in the following situations:

The user wants to build a multimodal vector retrieval service in ByteHouse.
The user needs to vectorize existing multimodal data (text, images, videos) and write it to ByteHouse.
The user mentions keywords such as "vector retrieval", "multimodal retrieval", "hybrid retrieval", "text-to-image search", "image-to-video search".

Prerequisites

Python 3.8+
Python dependencies:clickhouse-connect、clickhouse-driver(TCP scenario),volcengine-python-sdk[ark]、numpy
```
pip install clickhouse-connect clickhouse-driver "volcengine-python-sdk[ark]" numpy
```
jq（export_config.sh dependency, parses JSON configuration. Installation method:brew install jq or sudo apt install jq）
Valid ByteHouse connection information and Ark API Key (saved in ~/.bytehouse_config.json）

📁 File Description

SKILL.md — this file, the main Skill document.
scripts/embedding.py — multimodal vectorization module (MultimodalEmbedding）。
scripts/search_client.py — ByteHouse multimodal retrieval client (ByteHouseMultimodalSearch）。
scripts/examples.py — common call examples (initialization, table creation, writing, retrieval).
scripts/init.py — Python package entry point, exposing MultimodalEmbedding and ByteHouseMultimodalSearch。
scripts/export_config.sh — from ~/.bytehouse_config.json load configuration into environment variables (depends on jq）。

🧭 Agent Usage Guide

Confirm configuration：source scripts/export_config.sh, so that BYTEHOUSE_*、BH_ARK_* and other environment variables are ready.
Confirm scenario: clarify with the user whether to "create table + write" or "only perform retrieval", as well as the data modality (plain text / text + images / videos, etc.).
Choose index strategy：
- Data volume < 1 million:HNSW + COSINE(default).
- Larger data volume or wanting to save more memory:HNSW_SQ / IVF_FLAT / IVF_PQ。
Table creation + writing: call ByteHouseMultimodalSearch.create_multimodal_table(...)、insert_document(...) or batch_insert(...)。
Retrieval: prefer using vector_search(...) or hybrid_search(...), returned results automatically filter out zero vectors and abnormal data.
Security reminder：BH_ARK_API_KEY、BYTEHOUSE_PASSWORD and other credentials are only written to ~/.bytehouse_config.json, do not print them to logs or messages.

Configuration Description

Configuration is saved in ~/.bytehouse_config.json. If the file exists and is non-empty, use it directly; otherwise, ask the user to provide ByteHouse connection information and Ark API Key (you can ByteHouse Connection Information Retrieval Guide Send it to the user as well), and once obtained, write it to a JSON file to avoid asking repeatedly. When the user switches ByteHouse clusters or Ark accounts, update this file as well.

{
   "BYTEHOUSE_HOST": "<ByteHouse-host>",
   "BYTEHOUSE_PORT": "8123",
   "BYTEHOUSE_USER": "bytehouse",
   "BYTEHOUSE_PASSWORD": "<ByteHouse-password>",
   "BYTEHOUSE_DATABASE": "default",
   "BH_ARK_API_KEY": "<火山引擎方舟 API 密钥>",
   "BH_ARK_BASE_URL": "https://ark.cn-beijing.volces.com/api/v3",
   "BH_EMBEDDING_MODEL": "doubao-embedding-vision-251215",
   "EMBEDDING_DIMENSIONS": "2048"
}

BYTEHOUSE_HOST(host address) and BYTEHOUSE_PASSWORD(password)must be provided by the user。
It is recommended to keep the default values for other fields.
Before running, you mustexport the configuration to environment variables via source scripts/export_config.sh :search_client.py and embedding.py only reads configuration from environment variables.

source scripts/export_config.sh

🚀 Quick Start

1. Initialize the client

import sys, os
# 把 scripts/ 目录加入 sys.path
SKILL_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "scripts")
sys.path.insert(0, SKILL_SCRIPTS_DIR)

from search_client import ByteHouseMultimodalSearch

search = ByteHouseMultimodalSearch(connection_type="http")

2. Create a multimodal retrieval table

search.create_multimodal_table(
    table_name="multimodal_index",
    enable_text_search=True,
    index_type="HNSW",
    metric="COSINE",
)

3. Write data

# 文本
search.insert_document(
    "multimodal_index",
    doc_id=1,
    content_type="text",
    content="ByteHouse 是火山引擎推出的云原生数据仓库",
    title="ByteHouse 介绍",
    metadata={"category": "文档"},
)

# 图片（自动生成向量）
search.insert_document(
    "multimodal_index",
    doc_id=2,
    content_type="image",
    content="https://example.com/image.jpg",
    title="示例图片",
)

4. Vector retrieval

# 文本查询 → 向量检索
query_embedding = search.embedding.encode_text("云原生数据仓库")
results = search.vector_search(
    "multimodal_index",
    query_embedding=query_embedding,
    top_k=10,
)

# 混合检索（关键词 + 向量）
results = search.hybrid_search(
    "multimodal_index",
    query_text="云原生数据仓库",
    top_k=10,
)

For more examples, see scripts/examples.py。

Supported index types

Index Type	Description	Parameters
HNSW	High-performance general-purpose index	M, EF_CONSTRUCTION
HNSW_SQ	HNSW scalar quantization, saves memory	M, EF_CONSTRUCTION
IVF_FLAT	Inverted file index	dim, metric
IVF_PQ	Product quantization	dim, metric
IVF_PQ_FS	PQ fast search	dim, metric

Supported vectorization inputs

Text: any string.
Image: URL or base64 encoding.
Video: URL or base64 encoding (frame sampling + vectorization).

Failure Fallback Suggestions

Symptom	Handling Method
`ModuleNotFoundError: volcenginesdkarkruntime`	Ask the user to install `pip install "volcengine-python-sdk[ark]"`。
`BH_ARK_API_KEY` Not set	Guide the user to apply for an API Key in the Ark console and write it to `~/.bytehouse_config.json`。
`BYTEHOUSE_HOST` / `BYTEHOUSE_PASSWORD` Not set	Guide the user to provide the connection information, write it to `~/.bytehouse_config.json` and then `source scripts/export_config.sh`。
Returns an empty vector / all-zero vector	The client has already filtered zero vectors; if it still occurs frequently, check the vectorization input format (whether the URL is accessible, whether the image is corrupted).
Table creation error "index type xxx not supported"	Ask the user to confirm the cluster version, or switch to `HNSW` as a fallback.

About this skill