What is Seed1.5-Embedding
Seed1.5-Embedding is the latest vector model released by ByteDance's Seed team, further trained based on Seed1.5 (Doubao-1.5-pro). The model has achieved Chinese and English SOTA results on the authoritative benchmark MTEB, and has also achieved excellent results on the BRIGHT benchmark for reasoning-intensive retrieval tasks. The model adopts a Siamese dual-tower structure, relies on the Seed1.5 pre-trained LLM, and strengthens general representation capabilities through two-stage training. The first stage uses unsupervised data for pre-fine-tuning, transforming the generative model into an encoding model through contrastive learning; the second stage uses supervised data and synthetic data for fine-tuning, mixing multiple task data for multi-task optimization. Through strategies such as iterative hard negative mining, pseudo-negative filtering, and synthetic data, the data composition and quality are optimized to improve the model's performance in retrieval tasks. Seed1.5-Embedding supports multiple vector dimensions, including 2048, 1024, 512, and 256.
Main features of Seed1.5-Embedding
- Text semantic encoding: Encodes the semantics of input text into representation vectors in a high-dimensional space, making the vector similarity between related texts higher. This encoding method can support downstream tasks such as retrieval, classification, and clustering, and is widely used in scenarios such as search, recommendation, and content understanding.
- Retrieval tasks: Through vector similarity calculation, it quickly locates the information most relevant to the user query from massive documents. The model performs excellently on reasoning-intensive retrieval tasks and can understand complex query and document matching relationships.
- Multi-task optimization: Supports multiple tasks, including classification, clustering, pairwise classification, reranking, retrieval, and semantic textual similarity (STS) tasks, and is suitable for various application scenarios.
- Flexible vector dimension support: Supports multiple vector dimensions (2048, 1024, 512, 256). Users can choose the appropriate dimension according to specific needs. Even at lower dimensions, the model performance degradation is very small, providing flexible choices for storage and operational efficiency.
- Reasoning capability optimization: By constructing reasoning-intensive retrieval data, the model's reasoning capability in complex query and document matching is optimized, enabling it to handle more complex semantic relationships and logical reasoning tasks.
Technical principles of Seed1.5-Embedding
- Model architecture: Seed1.5-Embedding adopts a Siamese dual-tower vector model structure, and the vectors of queries and documents are matched by cosine similarity calculation. The model relies on the Seed1.5 pre-trained LLM, changes unidirectional attention to bidirectional, and builds a small-scale MoE (Mixture of Experts) model. The model parameters on the query side and document side are shared, ensuring high operational efficiency.
- Two-stage training process
- First stage: Uses unsupervised data for pre-fine-tuning, transforming a generative model with unidirectional Attention into an encoding model with bidirectional Attention through contrastive learning, fully modeling various text matching patterns.
- Second stage: Uses supervised data and synthetic data for fine-tuning, mixing multiple task data for multi-task optimization, allowing the model to learn the best representation patterns for each task.
- Data engineering strategies
- Negative example mining: Designs an iterative hard negative mining strategy, mining hard negatives according to the model's own preferences, improving the model's fine-grained relevance discrimination capability.
- Pseudo-negative filtering: Automatically filter texts that are too similar to positive examples to avoid false negatives affecting learning.
- Synthetic Data: Construct data for general scenarios and reasoning-intensive scenarios to improve the model's performance on complex retrieval tasks.
Project address of Seed1.5-Embedding
- HuggingFace Model Library：https://huggingface.co/ByteDance-Seed/Seed1.5-Embedding
Application scenarios of Seed1.5-Embedding
- Information Retrieval and Semantic Search: Seed1.5-Embedding can vectorize documents or web pages, support semantic-level search, and significantly improve recall and precision. For example, in scenarios such as question answering systems (QA), enterprise internal document retrieval, and customer service, the model can more accurately understand user query intent and quickly locate relevant documents.
- Text Clustering and Topic Identification: Using text vectors, Seed1.5-Embedding can cluster massive documents and automatically identify different topics and classification information.
- Recommendation Systems: In recommendation systems, the model can vectorize text information such as user reviews and product descriptions, and by calculating similarity, enable retrieval of similar products and similar users.
- Text Classification and Sentiment Analysis: Seed1.5-Embedding can generate vectors from text, which are then input into downstream classification models to improve the performance of tasks such as text classification, sentiment analysis, and stance analysis. Compared with traditional TF-IDF features, the generated vectors can more accurately express text meaning and contextual relationships.
- Complex Query Understanding and Reasoning: The model performs excellently on reasoning-intensive retrieval tasks and can deeply understand the deep matching relationships between complex queries and documents. For example, in complex search tasks in fields such as biology, Earth science, and programming, Seed1.5-Embedding can provide more accurate retrieval results.