infinity

Infinity is a high-throughput, low-latency inference serving engine focused on the deployment of tasks such as text embeddings, reranking models, CLIP, CLAP, and ColPali. The project is developed under the MIT license and aims to provide a fully featured inference server for embeddings, reranking, and related RAG tasks.

Key Capabilities

• Deploy HuggingFace models: Supports deploying any embedding, reranking, CLIP, and sentence transformer models.
• Fast inference backend: Built on PyTorch, optimum (ONNX/TensorRT), and CTranslate2, using FlashAttention to fully utilize NVIDIA CUDA, AMD ROCM, CPU, AWS INF2, or Apple MPS accelerators. Infinity dynamically batches and tokenizes in worker threads.
• Multimodal and multi-model: Multiple models can be mixed and matched, orchestrated by Infinity.
• Tested implementation: Includes unit tests and end-to-end tests to ensure embedding correctness.
• Easy to use: Built on FastAPI, provides Infinity CLI v2, supports startup via environment variables or arguments. OpenAPI aligns with OpenAI's API specification.

Supported Tasks and Models

Infinity supports a variety of tasks, including:

• Text embeddings: such as mixedbread-ai/mxbai-embed-large-v1, BAAI/bge-base-en-v1.5, etc.
• Reranking: such as mixedbread-ai/mxbai-rerank-xsmall-v1, BAAI/bge-reranker-base, etc.
• Image-text models: such as jinaai/jina-clip-v1, google/siglip-so400m-patch14-384, etc., which can be used for image library search.
• Audio-text models: such as LAION's CLAP model, which can be used to find music based on text descriptions.
• ColBERT and ColPali models: such as colbert-ir/colbertv2.0, vidore/colpali-v1.2-merged, etc., supporting late interaction embeddings.
• Text classification: such as ProsusAI/finbert, SamLowe/roberta-base-go_emotions, etc.

Usage

• Start via CLI: After installing with pip, run infinity_emb v2 --model-id BAAI/bge-base-en-v1.5. You can also use the prebuilt Docker container michaelf34/infinity, which requires mounting the accelerator.
• Multi-model startup: CLI v2 supports starting multiple models simultaneously, for example v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4, or via environment variables such as INFINITY_MODEL_ID="model/id1;model/id2;" and INFINITY_BATCH_SIZE="8;4;".
• Python API: Provides an asyncio-based Python interface, supporting asynchronous concurrent request processing.
• Python client: Provides generated client code, which can be installed via pip install infinity_client to call a remote Infinity instance.
• REST API: After startup, Swagger UI can be accessed at {url}:{port}/docs , for example http://localhost:7997/docs。

Use Cases

• Scenarios requiring private deployment of embedding services, similar to a private version of OpenAI text embeddings.
• Applications such as search, clustering, and recommendation systems.
• Image library search (finding related images via images or text).
• Music search (finding music via text descriptions).
• Classification tasks such as sentiment analysis and emotion detection.

Limitations

• For --engine ctranslate2, only BERT models are supported.
• For ColPali/ColQwen models, LoRA adapters are not supported; only "merged" models are supported.
• Open-source organizations for audio-text models are limited, and the sampling rate of audio data must match the model.
• Reranking models must be BERT-style single-class classification models.
• Specific information such as performance, fees, pricing, platform, and results is not yet provided.