SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsdeepseek-r1-vision

deepseek-r1-vision

deepseek-r1-vision is an open-source GitHub project that explores how to enable a vision-language model (VLM) to acquire deep reasoning capabilities similar to DeepSeek-R1 at low cost. The core idea of the project is to convert the original SFT question-answer data into COT (chain-of-thought) data that includes a "thinking" process, and then use SFT fine-tuning to "guide" the model to learn to think, requiring only limited GPU resources. The repository contains scripts for processing image-text conversations, plain text conversations, and modifying reasoning content to inject thinking elements, and also provides companion model download links and llama-factory fine-tuning configurations.

Open website

About this tool

deepseek-r1-vision is a method project that explores how to enable a vision-language model (VLM) to have deep thinking and reasoning capabilities similar to DeepSeek-R1 at low cost. The project name is the repository name, hosted on GitHub.

Purpose

The project aims to use limited GPU resources to enable a VLM to acquire deep thinking and reasoning capabilities similar to DeepSeek-R1 at low cost. Its core approach is to specially process supervised fine-tuning (SFT) data, converting the original question-answer pairs into chain-of-thought (COT) data that includes a "thinking" process, and then train the vision model using SFT, thereby "guiding" the model to learn how to think.

Main Capabilities

  • Convert original SFT data into a COT data structure with a thinking process, giving the data a "thinking" element.
  • Support processing of image-text conversation data.
  • Support processing of plain text conversations.
  • Provide configurations and scripts for SFT training using llama-factory.
  • Can perform full-parameter fine-tuning of the Qwen2-VL model, and continue training with COT data on domain datasets.

Usage

The project file structure is as follows:

  • DistilabelPipelineImage.py: processes conversations with image data.
  • DistilabelPipeline.py: processes plain text conversations.
  • ModifyThinkingContent.py: responsible for modifying reasoning content containing images, giving SFT data a "thinking" element, and is an important step in constructing image-text conversation COT data.
  • data/: data storage directory.

Model acquisition: The models used in the project can be obtained through the link https://modelscope.cn/models/atlas9999/Deepseek-r1-distill-qwen2_vl/summary Download, and the downloaded model files need to be placed in the specified location in the project for subsequent training and inference calls.

Training method: The project uses llama-factory for SFT training. Full-parameter fine-tuning requires reference to llama-factory, and the parameter scripts are located at ./qwen2vl_full_sft.yaml and qwen2vl_full_sft.yaml (note: the original text lists two identical file names). The project first uses a domain dataset to perform SFT fine-tuning on the qwen2-vl model, and then continues training using COT data (the method of this repository).

Reference repository:https://github.com/sungatetop/geology-lvlm.git

Notes: You need to pull the latest version of llama-factory and copy the .yaml file to the corresponding directory in llama-factory.

Applicable Scenarios

Suitable for developers who want to research or implement deep reasoning capabilities of vision-language models with limited GPU resources, especially scenarios interested in constructing and fine-tuning chain-of-thought data in image-text conversation settings.

Limitations

The materials do not provide details such as the specific supported model versions, minimum hardware requirements, training data scale, performance evaluation, fee or price information, or platform compatibility. This information is not currently provided.

Related tools

Browse category