What is Vary-toy
Vary-toy is a small vision-language model (LVLM) jointly proposed by researchers from Megvii, UCAS, and HUST, aiming to address the challenges of training and deploying large vision-language models (LVLMs). For researchers with limited resources, large models typically have billions of parameters and are difficult to train and deploy on consumer-grade GPUs (such as GTX 1080Ti). The core goal of Vary-toy is to enable researchers to experience all the functionalities of current LVLMs (document OCR, visual grounding, image captioning, visual question answering, etc.) under limited hardware resources.
Main Features of Vary-toy
- Document-level Optical Character Recognition (OCR): Vary-toy can recognize and understand text in document images, which is very useful in scenarios such as processing scanned documents and PDF files.
- Image Captioning: The model can generate descriptive text for images, which is crucial for understanding image content and image captioning tasks (such as VQA).
- Visual Question Answering (VQA): Vary-toy can answer questions about image content, which involves understanding the visual information of the image as well as the text information related to it.
- Object Detection: Through enhanced visual vocabulary, Vary-toy possesses the ability of natural object perception (grounding), and can identify and locate objects in images.
- Image-to-Text Conversion: Vary-toy can convert image content into structured text formats, such as converting PDF images into Markdown format.
- Multimodal Dialogue: The Vary-toy model supports multimodal dialogue, and can understand and generate dialogue related to image content.
Official Entrances of Vary-toy
- Official project homepage:https://varytoy.github.io/
- Arxiv research paper:https://arxiv.org/abs/2401.12503
- Demo running address:https://vary.xiaomy.net/
- GitHub code repository:https://github.com/Ucas-HaoranWei/Vary-toy
Technical Principles of Vary-toy
The working principle of Vary-toy is based on several key technologies and design decisions, which together improve the model's performance on vision-language tasks while keeping the model small. The following are the main components of Vary-toy's working principle:
- Visual Vocabulary Generation: Vary-toy utilizes a small autoregressive model (OPT-125M) to generate a new visual vocabulary network. This network learns how to effectively encode visual information by processing PDF image-text pairs and object detection data in natural images. Compared with the traditional Vary model, when generating visual vocabulary, Vary-toy no longer treats natural images as negative samples, but instead treats them as positive samples, thereby making fuller use of the network's capacity.
- Fusion of Visual Vocabulary and CLIP: After generating the new visual vocabulary, Vary-toy combines it with the original CLIP (Contrastive Language-Image Pre-training) model. CLIP is a powerful vision-language model that can map images and text to a shared embedding space. In this way, Vary-toy can leverage CLIP's image understanding ability while enhancing the processing of text information through the new visual vocabulary network.
- Multi-task Pre-training: Vary-toy adopts a multi-task learning strategy during the pre-training stage, which means that the model simultaneously processes multiple types of data during training, such as image captioning, PDF OCR, object detection, pure text dialogue, and visual question answering (VQA). This multi-task training helps the model learn richer visual and language representations and improve its generalization ability on various downstream tasks.
- Model Architecture: Vary-toy follows Vary's pipeline design, but with some structural adjustments. When an image is input, the new visual vocabulary branch resizes the image to a resolution of 1024×1024, while the CLIP branch obtains a 224×224 image through center cropping. The image features output by the two branches are merged and fed as input into the 1.8B-parameter Qwen-1.8B language model.
- Data Input Format: To adapt to different tasks, Vary-toy needs to handle multiple input formats. For example, for PDF image-text pairs, the model uses a specific prompt (such as "Provide the OCR results of this image.") to guide the output toward the correct result. For object detection tasks, the model uses different prompt templates to handle multiple objects in the image.
- Fine-Tuning (SFT): After pretraining, Vary-toy further optimizes the model through an instruction tuning (SFT) stage. This stage uses the LLaVA-80K dataset, an image dataset containing detailed descriptions and prompts, generated by GPT4. This helps the model better understand and generate text related to image content.
Through these working mechanisms, Vary-toy can effectively handle complex vision-language tasks while keeping the model compact. This design makes Vary-toy a powerful tool for vision-language research in resource-constrained environments.
How to Use Vary-toy
- Visit Vary-toy's official demo experience address (vary.xiaomy.net）
- Click to upload an image or select a sample image on the left
- Enter a prompt instruction such as describing the image content, detecting objects in the image, etc.
- Wait for the model to generate the result