clip-retrieval is an open-source project for easily computing CLIP embeddings and building retrieval systems based on these embeddings. It aims to enable users to quickly set up semantic search systems, supporting multimodal retrieval of text and images.
Purpose
clip-retrieval's core purpose is to build semantic search systems based on CLIP embeddings. It can convert images and text into CLIP embedding vectors, then build efficient indexes, thereby enabling functions such as text-to-image search, image-to-image search, and querying by embedding. This project is particularly suitable for scenarios that need to handle large-scale image-text datasets, such as preparing data for machine learning, building multimodal retrieval applications, etc.
Main capabilities
- Embedding computation: The clip inference component can quickly compute CLIP embeddings of images and text, with speeds reaching 1500 samples/second on a 3080 GPU. It supports multiple CLIP models, including open_clip and Hugging Face models, and also supports multilingual CLIP (MCLIP).
- Index building: clip index uses autofaiss to build efficient FAISS indexes from embeddings, and can scale to hundreds of millions of samples.
- Data filtering: clip filter allows filtering data by query using a CLIP index, for example extracting the 100 images that best match "cat".
- Backend service: clip back provides a simple Flask service to host the index, supporting options such as memory mapping, HDF5/Arrow caching, safety models, violence detection, aesthetic scoring, etc.
- Frontend interface: clip front is a simple UI for querying the backend, viewable at clip-retrieval ui.
- End-to-end workflow: clip end2end runs img2dataset, inference, indexing, backend, and frontend, simplifying the entire workflow.
- Remote querying: ClipClient allows remote querying of a clip-retrieval backend via Python, supporting parameters such as specifying the index name, aesthetic score, multilingual, modality, number of results, deduplication, safety model, etc.
- Distributed inference: Supports distributed inference across multiple nodes and multiple GPUs, and also supports SLURM clusters.
- Storage support: Supports loading/writing webdataset-format data from HDFS and S3.
- DeepSparse support: Fast sparse model inference on CPU via DeepSparse.
Usage
Installation:pip install clip-retrieval
Compute embeddings
Use the clip-retrieval inference --input_dataset image_folder --output_folder embeddings_folder command. The input can be a folder containing images (optionally with text files of the same name), and the output includes numpy files and parquet files containing image embeddings, text embeddings, and metadata.
Build an index
Use the clip-retrieval index --embeddings_folder embeddings_folder --index_folder index_folder command, with configurable parameters such as memory usage, subfolders, metadata copying, number of threads, etc. The output includes an image index, a text index, and a metadata folder.
Filter data
Use the clip-retrieval filter --query "cat" --output_folder "cat/" --indice_folder "indice_folder" command to copy the best-matching images to the output folder.
Start the backend
Use the clip-retrieval back --indice_folder indice_folder command, with configurable options such as JIT, CLIP model, multilingual options, returned columns, memory mapping, HDF5/Arrow caching, safety models, violence detection, aesthetic embeddings, etc.
End to end
Use the clip-retrieval end2end command to automatically run img2dataset, inference, indexing, backend, and frontend.
Remote querying
When initializing ClipClient, specify parameters such as backend_url and indice_name, then call the query method.
Distributed inference
Refer to the docs/distributed_clip_inference.md tutorial.
HDFS/S3 support
Via --input_dataset "pipe:hdfs dfs -cat ..." or --input_dataset "pipe:aws s3 cp ..." and other methods; the output can specify an hdfs:// or s3:// path.
Use cases
- Building large-scale image-text semantic search systems.
- Preprocessing and augmentation for machine learning datasets, for example generating text anchors for images.
- Fast similarity search in datasets with millions of records.
- Multilingual image retrieval.
- Retrieval applications that require safety filtering (NSFW, violence).
- Academic research or industrial-grade multimodal retrieval prototype development.
Limitations
- The project documentation does not clearly state the specific billing model, pricing, or commercial licensing information, and this is not currently provided.
- Performance data is based on specific hardware (3080 GPU), and actual performance may vary depending on the hardware configuration.
- Large-scale indexing requires sufficient memory and storage resources; index memory usage is configurable but affects recall.
- Some advanced features (such as safety models and violence detection) require additional dependency installation.
- The project does not provide detailed information about platform compatibility, and this is not currently provided.
