What is MEXMA
MEXMA is a novel pretrained cross-lingual sentence encoder introduced by Meta AI. It improves the quality of sentence representations by combining sentence-level and word-level objectives. During training, MEXMA uses the sentence representation of one language to predict masked words in another language, supporting direct updates to both the sentence representation and all words in the encoder. MEXMA demonstrates superior performance on multiple tasks, surpassing existing pretrained cross-lingual sentence encoders such as LaBSE and SONAR. MEXMA supports 80 languages and performs excellently in downstream tasks such as sentence classification.
Main Features of MEXMA
- Cross-Lingual Sentence Encoding: MEXMA encodes sentences from different languages into fixed-size vectors, which are compared and manipulated in a shared multilingual space.
- Combination of Sentence-Level and Word-Level Objectives: By simultaneously considering the overall meaning of the sentence and the contribution of each word in the sentence, MEXMA improves the quality and alignment of sentence representations.
- Improved Multi-Task Performance: MEXMA performs excellently in a variety of downstream tasks, including sentence classification, text mining, and semantic textual similarity tasks.
- Support for 80 Languages: MEXMA supports up to 80 languages, making it suitable for a wide range of multilingual application scenarios.
Technical Principles of MEXMA
- Combination of Sentence-Level and Word-Level Objectives: MEXMA trains the model with sentence-level objectives and introduces word-level objectives. During training, the model learns the overall representation of the sentence as well as the representation of each word in the sentence.
- Cross-Lingual Masking Task: MEXMA uses the sentence representation of one language to predict masked words in another language. The cross-lingual masking task forces the model to learn sentence representations that capture the core information of the sentence, ensuring that information is alignable across different languages.
- Direct Encoder Updates: In MEXMA, the sentence representation can update the encoder, and the representation of each word directly updates the encoder. The bidirectional update mechanism enables the model to more effectively learn high-quality sentence and word representations.
- Symmetric Architecture: MEXMA is based on a symmetric architecture that simultaneously masks and predicts sentences in two languages, ensuring that the model is balanced between the two languages and can generate two clean sentence vectors, which is crucial for alignment between languages.
- Non-Contrastive Loss Function: To avoid model collapse and improve the quality and alignment of representations, MEXMA uses non-contrastive loss functions, such as mean squared error (MSE) loss, to strengthen the spatial proximity of semantically equivalent sentences across different languages.
MEXMA Project Address
- GitHub Repository：https://github.com/facebookresearch/mexma
- HuggingFace Model Library：https://huggingface.co/facebook/MEXMA
- arXiv Technical Paper：https://arxiv.org/pdf/2409.12737
Application Scenarios of MEXMA
- Cross-Lingual Information Retrieval: Use MEXMA to convert documents in different languages into the same embedding space for cross-lingual search and retrieval.
- Machine Translation: MEXMA improves machine translation systems by providing more accurate cross-lingual sentence representations to enhance translation quality.
- Multilingual Text Classification: When processing multilingual text data, MEXMA can classify text content, such as sentiment analysis, topic classification, etc.
- Semantic Textual Similarity Evaluation: MEXMA evaluates the similarity between sentences in different languages, suitable for comparing and matching similar text content.
- Cross-lingual question answering system: In multilingual question answering systems, MEXMA helps understand questions in different languages and find answers in the corresponding language.