WeMM-Embedding - WeChat Open-Source General Multimodal Embedding Model

What is WeMM-Embedding

WeMM-Embedding is a general multimodal Embedding model family open-sourced by Tencent WeChat Vision Team, offering three specifications: 2B/4B/9B. The model uniformly encodes text, images, videos, visual documents, and interleaved multimodal inputs into a shared vector space, ranks first on the MMEB-v2 benchmark leaderboard, and has been deployed at large scale in WeChat Channels, Official Accounts, Search, and other recommendation and retrieval businesses.

Main Features of WeMM-Embedding

• Unified Multimodal Encoding: Supports text, images, videos, visual documents, and arbitrarily interleaved multimodal inputs, uniformly mapped into a shared dense vector space.
• Flexible Dimension Output: Based on Matryoshka representation learning, a single inference can output nested vectors from 64 to 4096 dimensions, and 256 dimensions can retain nearly 99% of the full-dimensional performance.
• Multi-Task Adaptation: Covers scenarios such as retrieval, classification, question answering, visual grounding, cross-domain matching, Agent tool/GUI/memory retrieval, and more.
• Efficient Inference Deployment: Supports multiple frameworks such as Transformers, Sentence Transformers, vLLM, and SGLang, making it easy to integrate into production environments.

Technical Principles of WeMM-Embedding

• Architecture Design: Uses the Qwen3.5 multimodal large language model as the backbone, by appending at the end of the input sequence <embedding> special token, taking its final-layer hidden state and applying L2 normalization to obtain the output representation, naturally supporting arbitrarily interleaved multimodal inputs under causal attention.
• Two-Stage Training Strategy: The first stage uses hundreds of millions of heterogeneous pairs to establish a general multimodal alignment space; the second stage performs fine-grained refinement on curated data by combining hard negative samples, reranker supervision, and cross-scale Embedding distillation, achieving progressive improvement from broad coverage to high precision.
• Data Construction and Curation: Uses a unified Pair-Based format to integrate six major categories of supervision data; balances semantic distribution through Semantic-ID-guided resampling, uses multimodal large models to automatically filter low-quality samples and correct text errors, and separately constructs explicit hard negatives for text, images, and videos to enhance discriminative ability.
• Training Objective System: Standard paired data uses InfoNCE contrastive learning and introduces duplicate-aware masking to avoid false negatives; graded relevance data uses a score-gap-weighted CoSENT ranking loss; Stage 2 introduces reranker scores to provide task-level fine-grained supervision, using the 9B model as a teacher to distill soft similarity distributions to smaller models via bidirectional KL divergence.

Follow WeChat and reply "Open Source" to jointhe AI Open-Source Project Communication Group

How to Use WeMM-Embedding

• Environment Preparation: Clone the GitHub repository and run pip install -r requirements.txt to install dependencies; it is recommended to use transformers==5.2.0 to ensure reproducible results.
• Obtain the Model: Download from Hugging Face WeMM-Embedding-2B/4B/9B model weights to local.
• Transformers Inference: Run examples/transformers_inference.py, pass in the model path and text/image/video paths, and specify --dimension to obtain the Embedding of the corresponding dimension.
• Sentence Transformers Inference: Run examples/sentence_transformers_inference.py, you can directly use a Hugging Face model ID, and through encode() uniformly encode multimodal inputs.
• Matryoshka dimensionality reduction: Take the first d dimensions of the full-dimensional vector, then use torch.nn.functional.normalize to reapply L2 normalization, and you can obtain a low-dimensional representation.
• Service-oriented deployment: In production environments, use vLLM (--runner pooling) or SGLang (--is-embedding) to launch the inference service, supporting high-concurrency calls.

Core advantages of WeMM-Embedding

• Extreme parameter efficiency: A 2B model surpasses the previous leading 8B open-source baseline, redefining the performance-efficiency boundary of multimodal Embedding.
• Truly unified multimodal representation: Natively supports text, images, videos, visual documents, and arbitrary interleaved combinations of inputs, with no need to design separate encoding pathways for different modalities.
• Flexible Matryoshka dimensions: A single inference can output vectors from 64 to 4096 dimensions, and 256 dimensions can still retain nearly 99% of the full-dimensional performance, significantly reducing storage and retrieval costs.
• Validated through large-scale business operations: It has achieved consistent improvements in 14 A/B tests across core businesses such as WeChat Channels, Official Accounts, and Search, and has been fully launched in the production environment.
• Advanced progressive training: Through a two-stage strategy of “large-scale alignment + curated distillation,” combined with Semantic-ID data curation and cross-scale knowledge distillation, it achieves both broad coverage and high precision.

Project address of WeMM-Embedding

• GitHub repository：https://github.com/Tencent/WeMM-Embedding
• HuggingFace model library：https://huggingface.co/collections/tencent/wemm-embedding
• arXiv technical paper：https://arxiv.org/pdf/2608.24053

Comparison of WeMM-Embedding with similar competing products

Application scenarios of WeMM-Embedding

• Cross-modal semantic retrieval: The model supports cross-modal retrieval in any direction, such as text-to-image search, image-to-text search, and text-to-video search, and is suitable for massive material library lookup and copyright monitoring on content platforms.
• Personalized recommendation systems: Encode image-text and video content into unified vectors for candidate recall, user interest sequence modeling, and cross-domain content matching, and it has already been implemented in WeChat Channels, Official Accounts, and e-commerce recommendation scenarios.
• Visual Document Understanding and Retrieval: Vectorize visual documents such as PDFs, charts, invoices, and screenshots to support intelligent document question answering, ticket archive retrieval, and enterprise knowledge base construction.
• AI Agent Tools and Memory Retrieval: Provide Agents with tool invocation retrieval, GUI interface element localization, and long-range memory retrieval capabilities, supporting complex task planning and multi-step interaction.
• Multimodal Content Classification and Moderation: Map image and video content into a semantic space to enable object recognition, scene classification, action detection, and similarity comparison and cluster analysis of violating content.