byted-bytehouse-hybrid-search

ByteHouse Hybrid Search Skill

🚀 Quick Start

Environment Preparation

pip install clickhouse-connect volcengine-python-sdk[ark] numpy scipy

Configuration Instructions

The configuration is saved in ~/.bytehouse_config.json . If this file exists and is not empty, use the configuration from the file directly. If it does not exist, ask the user to provide the ByteHouse connection information (also send this document to the customer; the document explains how to obtain the host address and password:https://www.volcengine.com/docs/6517/1121919?lang=zh ). After the user provides the information, save it to the json file to avoid repeatedly requesting connection information from the user. When the user switches ByteHouse clusters, modify this file as well.

{
   "BYTEHOUSE_HOST": "<ByteHouse-host>",
   "BYTEHOUSE_PORT": "8123",
   "BYTEHOUSE_USER": "bytehouse",
   "BYTEHOUSE_PASSWORD": "<ByteHouse-password>",
   "BYTEHOUSE_SECURE": true,
   "BYTEHOUSE_VERIFY": true, 
   "BH_ARK_API_KEY": "<火山引擎方舟API密钥>",
   "BH_ARK_BASE_URL": "https://ark.cn-beijing.volces.com/api/v3",
   "BH_EMBEDDING_MODEL": "doubao-embedding-vision-251215"
}

Among them, BYTEHOUSE_HOST (host address) and BYTEHOUSE_PASSWORD (password)must beprovided by the user. BH_ARK_API_KEY is an optional configuration, used only for embedding, and can be ignored by the user on first use. The remaining configurations are fixed.

Execute scripts/export_config.sh to import the configuration information into environment variables

source scripts/export_config.sh

📚 Core Capabilities

1. Text Vectorization

Generates text vectors based on the Doubao text vectorization model, supporting Chinese text of any length.

2. Dual Index Construction

Index Type	Description	Applicable Scenarios
Full-text inverted index	Full-text search based on the BM25 algorithm, supporting keyword matching	Precise keyword recall
Vector index	Vector similarity search based on HNSW, supporting semantic matching	Semantic similarity recall

3. Core Features

Feature	Method	Description
Full-text search	`fulltext_search()`	Full-text search based on BM25, returning BM25 scores
Vector search	`vector_search()`	Vector search based on cosine similarity, returning similarity scores
Hybrid search + RRF reranking	`hybrid_search()`	After dual-path recall, use the RRF algorithm to rerank and return the final results
Automatically generate vectors	`insert_document()`/`batch_insert_documents()`	Automatically generate and store vectors when inserting documents, with no manual processing required
Single document vector update	`update_document_embedding()`	Regenerate and update the vector for a single document
Batch complete missing vectors	`batch_update_missing_embeddings()`	Automatically scan all documents in the table that are missing vectors, and generate and complete vectors in batches

4. RRF Reranking Algorithm

Reciprocal Rank Fusion algorithm, which combines the ranking results of full-text search and vector search, formula:

score = Σ 1 / (k + rank)

The default k=60, adjustable as needed.

📖 Code Implementation

The complete example code implementation is located in the scripts/ directory:

scripts/embedding.py - Text vectorization module
scripts/hybrid_search_client.py - ByteHouse hybrid search client
scripts/examples.py - Usage example
scripts/export_config.sh - Import the information from the configuration file into environment variables

Quick Usage

from scripts import ByteHouseHybridSearch

# 初始化客户端
search = ByteHouseHybridSearch(connection_type="http")

# 创建混合检索表（自动构建全文索引和向量索引）
search.create_hybrid_table("my_hybrid_index")

# 插入文档（自动生成向量 + 存储原始文本）
search.insert_document("my_hybrid_index", doc_id=1, 
                      title="ByteHouse 混合检索", 
                      content="ByteHouse 支持全文检索和向量检索，可实现混合检索能力")

# 混合检索（自动执行全文+向量检索，RRF重排返回结果）
results = search.hybrid_search("my_hybrid_index", query="ByteHouse检索能力", top_k=10)

⚙️ Best Practices

Table Creation Configuration

CREATE TABLE {table_name} (
    `doc_id` UInt64,
    `title` String,
    `content` String,
    `embedding` Array(Float32),
    -- 全文倒排索引（version=2支持BM25分数）
    INDEX content_idx content TYPE inverted('standard', '{"version":"v4"}') GRANULARITY 1,
    -- 向量索引（HNSW算法，余弦相似度）
    INDEX embedding_idx embedding TYPE HNSW_SQ('DIM={vec_dimensions}', 'metric=COSINE', 'M=32', 'EF_CONSTRUCTION=256') GRANULARITY 1
)
ENGINE = CnchMergeTree()
ORDER BY doc_id
SETTINGS 
    index_granularity = 1024,
    enable_vector_index_preload = 1

RRF Parameter Adjustment

When full-text search results are more important, you can lower therrf_kvalue (30-60 recommended)
When vector retrieval results are more important, you can increase therrf_kvalue (recommended 60-100)

About this skill