SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

KaLM-Embedding - A Series of Text Embedding Models Launched by Tencent

Author: AI小集 · Original source: AI工具集

What is KaLM-Embedding

KaLM-Embedding is a series of high-performance text embedding models launched by the Tencent team, improving text embedding performance through advanced training techniques and high-quality data. The latest version, KaLM-Embedding-V2, introduces multiple innovations in architecture and training methods, such as removing the causal attention mask to enable bidirectional representation learning and adopting a multi-stage training pipeline (including pretraining, fine-tuning, and contrastive distillation), significantly enhancing the model's generalization ability and semantic understanding. The latest KaLM-Embedding-Gemma3-12B-2511 is an important version in the series, based on a larger parameter scale (12B parameters), further optimizing model performance and suitable for complex tasks requiring higher precision.

KaLM-Embedding

Main Features of KaLM-Embedding

  • Efficient Text Embedding Generation: KaLM-Embedding can efficiently convert text into fixed-length embedding vectors, suitable for a variety of natural language processing tasks such as retrieval, classification, and semantic matching.
  • Multilingual and Cross-Lingual Capabilities: Supports multilingual text embedding, enabling semantic alignment and cross-lingual retrieval between different languages and improving performance on cross-lingual tasks.
  • Flexible Embedding Dimensions: Supports flexible-dimension embeddings and, through Matryoshka representation learning, maintains high performance at different dimensions, adapting to a variety of application scenarios.
  • Strong Downstream Task Adaptability: Suitable for a variety of downstream tasks, including text classification, semantic matching, information retrieval, and clustering analysis, providing comprehensive support for natural language processing.

Technical Principles of KaLM-Embedding

  • Bidirectional Attention Mechanism: Removes the traditional causal attention mask and adopts a bidirectional attention mechanism, enabling the model to consider contextual information simultaneously and improving the accuracy of semantic representation.
  • Mean Pooling: Uses a simple mean pooling method to convert text sequences into fixed-length embedding vectors, ensuring that the embeddings output by the model are suitable for a variety of downstream tasks.
  • Multi-Stage Training Pipeline: Combines three stages—pretraining, fine-tuning, and contrastive distillation—to progressively improve the model's embedding capability. The pretraining stage uses large-scale weakly supervised data, the fine-tuning stage uses high-quality supervised data, and the contrastive distillation stage learns fine-grained knowledge from a stronger teacher model.
  • Focal Reweighting Mechanism: Through focal-style reweighting, focuses on difficult-to-learn samples, optimizing the training process and improving the model's ability to learn from hard samples.
  • Online Hard Negative Mixing: Dynamically generates hard negative samples, continuously providing highly informative training signals, avoiding outdated negative samples, and enhancing the model's discriminative ability.
  • Matryoshka Representation Learning: Supports flexible-dimension embeddings and maintains high performance at different dimensions through Matryoshka techniques, adapting to a variety of application scenarios.
  • High-Quality Data Support: Uses diverse and high-quality datasets for training, combining techniques such as task-specific instructions, hard negative mining, and multi-category labeling to ensure embedding quality.
  • Contrastive Learning and Distillation: Uses the InfoNCE loss function for contrastive learning and learns fine-grained soft signals from a stronger teacher model through contrastive distillation, further optimizing model performance.
  • Temperature Coefficient Adjustment: Introduce a temperature coefficient in contrastive distillation, optimize the distribution of learning signals by adjusting the temperature coefficient, and improve the model's learning efficiency.
  • Flexible model architecture: Based on a compact model architecture (such as a 0.5B parameter scale), it achieves high-performance text embeddings while maintaining efficiency.

Model series versions of KaLM-Embedding

  • KaLM-Embedding-V1: It is the initial version of the series, based on a compact model architecture, uses a causal attention mask, and is mainly used for basic text embedding tasks.
  • KaLM-Embedding-V2: On the basis of V1, multiple improvements were made, the causal attention mask was removed to achieve bidirectional representation learning, and a multi-stage training pipeline was introduced, including pretraining, fine-tuning, and contrastive distillation, significantly improving model performance.
  • KaLM-Embedding-V2.5: It is a further optimized version of V2. Through contrastive distillation, it learns fine-grained soft signals from a stronger teacher model, further improving the model's embedding capability and generalization performance.
  • KaLM-Embedding-Gemma3-12B-2511: It is the latest version, based on a larger parameter scale (12B parameters), further optimizing model performance and suitable for complex tasks that require higher accuracy.

Project address of KaLM-Embedding

  • Project official website：https://kalm-embedding.github.io/
  • HuggingFace model library：https://huggingface.co/tencent/KaLM-Embedding-Gemma3-12B-2511
  • arXiv technical paper：https://arxiv.org/pdf/2506.20923

Application scenarios of KaLM-Embedding

  • Text classification: It can perform efficient classification processing on text, helping users quickly identify the topic and category of the text.
  • Semantic matching: It can accurately determine the semantic similarity between different texts and is widely used in fields such as search engines and recommendation systems.
  • Information clustering: It automatically categorizes similar text information, making it convenient for users to manage and analyze large amounts of text data.
  • Search recommendation: It improves the relevance of search results and the accuracy of recommendations through semantic understanding, providing users with a more personalized experience.
  • Multilingual understanding: It supports multilingual semantic alignment, performs well in tasks such as cross-language retrieval and translation, and can achieve more accurate cross-language semantic understanding.