SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsu-LLaVA

u-LLaVA

u-LLaVA is a multimodal multi-task unified framework based on large language models (LLMs), developed by OPPOMKLab, and has been accepted to the main track of ECAI 2024. The project aims to uniformly handle multiple vision-language tasks through LLMs, including image captioning, video captioning, visual question answering (VQA), referring expression segmentation (RES), salient object segmentation, semantic segmentation, referring expression comprehension (REC), and instance segmentation. u-LLaVA provides a complete training and evaluation pipeline, supports multi-stage training (pre-training and fine-tuning), and has released code, segmentation weights, and the Salient-15k dataset. The project uses the Apache License, requires installing dependencies, building GroundingDINO's CUDA kernel, and preparing relevant datasets. Training is divided into two stages: Stage I pre-training and Stage II fine-tuning, supporting multi-GPU and single-GPU training. u-LLaVA provides a unified solution for research and applications of multimodal tasks.

Open website

About this tool

u-LLaVA is a multimodal multi-task large language model framework designed to uniformly handle multiple vision-language tasks through LLMs. The project is developed by OPPOMKLab and has been accepted to the main track of ECAI 2024.

Purpose

The main purpose of u-LLaVA is to unify multimodal tasks, including visual understanding, visual segmentation, and visual grounding. Specific tasks include:

  • Visual understanding: image captioning, video captioning, visual question answering (VQA)
  • Visual segmentation: referring expression segmentation (RES), salient object segmentation, semantic segmentation, instance segmentation
  • Visual grounding: referring expression comprehension (REC)

Main Capabilities

  • Multimodal multi-task unification: handling multiple vision-language tasks through LLMs.
  • Support for multiple datasets: including mixed datasets of text, image-text, video-text, etc.
  • Training pipeline: divided into Stage I pre-training and Stage II fine-tuning, supporting multi-GPU and single-GPU training.
  • Evaluation: provides batch evaluation and qualitative inference capabilities.
  • Released resources: code, segmentation weights, Salient-15k dataset.

Usage

  1. Install dependencies:pip install -r requirements.txt
  2. Build GroundingDINO's CUDA kernel:cd ./models/GroundingDINO && ./install.sh && cd ../..
  3. Download annotations and image data: download the ullava-modified annotations, LLaVA pre-training and fine-tuning annotations, and image storage according to the provided links.
  4. Prepare the open-source LLaMA model: note that LLaMA2 uses bf16 training, and using fp16 in Stage I may cause convergence errors; Llama-2's tokenizer.legacy defaults to False, which may cause tokenization mismatch errors. The base LLM used in the paper is Vicuna-v1.1, not LLaMA2.
  5. Train Stage I: use multi-GPU or single-GPU commands, for example python train_ullava_core.py --cfg_path './configs/train/ullava_core_stage1.yaml'. Stage I uses 4 A100 80G and bf16 training for 1 epoch takes about 6 hours.
  6. Train Stage II: use multi-GPU or single-GPU commands, for example python train_ullava.py --cfg_path './configs/train/ullava_stage2_lora.yaml'.
  7. Evaluation: modify the parsers in evaluation/inference_ullava_core.py and evaluation/inference_ullava.py for batch evaluation and qualitative inference.

Applicable Scenarios

u-LLaVA is suitable for research and development of multimodal tasks, such as visual question answering, image/video captioning, image segmentation, and visual grounding, etc. It can help researchers and developers explore multimodal tasks within a unified framework.

Limitations

  • Training requires relatively high computational resources (such as A100 80G).
  • Some weights and features (such as grounding&segmentation weights, ViT-336 support, etc.) have not yet been released and are expected to be released before July or August 2024.
  • The license requirements of the original datasets must be followed.
  • The project uses the Apache License.

Related tools

Browse category