What is Maya
Maya is an open-source multilingual multimodal model based on instruction fine-tuning to extend the model's capabilities across multiple languages and cultural backgrounds. Maya is based on the LLaVA framework and includes a newly created pretraining dataset containing eight languages, improving cultural and linguistic understanding in vision-language tasks. Based on toxicity analysis and dataset filtering, Maya ensures the safety and quality of training data, supports multiple languages including Chinese, French, Spanish, Russian, Hindi, Japanese, and Arabic, and is committed to improving the quality of AI content generation for low-resource languages.
Maya's main features
- Multilingual support: Maya can process and understand eight different languages, including Chinese, French, Spanish, Russian, Hindi, Japanese, Arabic, and English, enhancing support for low-resource languages.
- Multimodal capability: Combining image and text data allows machines to understand the visual world through natural language and perform tasks such as image captioning and visual question answering.
- Instruction fine-tuning: Based on instruction fine-tuning, it better understands and responds to natural language instructions, improving performance and adaptability in real-world applications.
- Dataset creation and toxicity filtering: Create a multilingual image-text pretraining dataset and perform toxicity analysis and filtering to ensure data safety and quality.
- Cross-cultural understanding: Based on multilingual and multimodal data, it better understands and processes visual and linguistic information under different cultural backgrounds.
Maya's technical principles
- Model architecture: Based on the LLaVA 1.5 architecture, using the Aya-23 8B model as the multilingual language model (LLM) and SigLIP as the vision encoder, it supports multilingual and multimodal input.
- Pretraining dataset: Create a multilingual image-text pretraining dataset containing 558,000 images, covering eight languages and supporting the development of multilingual vision-language models.
- Toxicity analysis: Use LLaVAGuard 7B and Toxic-BERT to perform toxicity analysis on images and text in the dataset, identifying and filtering out unsafe or harmful content.
- Pretraining and fine-tuning：
- Pretraining: Use the projection matrix W to convert image features into language features, and perform pretraining based on multi-turn dialogue data to optimize the alignment of images and text.
- Fine-tuning: Perform fine-tuning on the PALO 150K instruction fine-tuning dataset to further improve the model's ability to understand and respond to instructions.
- Cross-modal alignment: Based on the projection matrix and training strategy, optimize the alignment between image features and language features, improving the model's performance in vision-language tasks.
Maya's project address
- GitHub repository：https://github.com/nahidalam/maya
- HuggingFace model library：https://huggingface.co/maya-multimodal/maya
- arXiv technical paper：https://arxiv.org/pdf/2412.07112
Maya's application scenarios
- Cross-language content understanding: Help users understand image content in different languages, for example, recognizing and interpreting road signs, advertisements, menus, etc. in multilingual environments.
- Image and video analysis: In fields such as security monitoring and content moderation, analyze images and videos to identify and filter inappropriate content.
- Education and learning: Provide image and text analysis of multilingual learning materials for non-native learners, enhancing the language learning experience.
- Tourism and navigation: Help tourists identify and translate street signs, maps, and cultural landmarks in different countries.
- E-commerce: On multilingual e-commerce platforms, help users understand product descriptions and images, improving the shopping experience.