ImageBind was completed by Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra, and others. The paper was published at CVPR 2023 and was selected as a Highlighted paper. The project provides a PyTorch implementation and pretrained models, and the official repository is https://github.com/facebookresearch/ImageBind .
Main capabilities ImageBind learns a joint embedding space across six modalities, covering images, text, audio, depth, thermal imaging, and IMU data. According to the materials, it supports out-of-the-box emergent applications, including cross-modal retrieval, arithmetic composition of modalities, cross-modal detection and generation, and emergent zero-shot classification performance.
Usage The project provides a PyTorch implementation. Before use, PyTorch 2.0+ and other third-party dependencies need to be installed. The repository provides examples of extracting and comparing features across modalities, such as comparing features of images, text, and audio. Windows users may need to install soundfile to read and write audio files. For specific installation and running steps, please refer to the documentation files in the repository.
Applicable scenarios According to the official description, ImageBind is suitable for research and application scenarios that require joint cross-modal representations, such as cross-modal retrieval, modality composition, cross-modal detection and generation, and zero-shot classification.
Limitations and license The ImageBind code and model weights are released under the CC-BY-NC 4.0 license, and use must comply with that license and the LICENSE file in the repository. For more details related to the model, please refer to the model card. Information on whether it can be used for commercial purposes, the specific scope of platform support, and pricing is not currently provided in the materials.
