SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsOcean-R1

Ocean-R1

Ocean-R1 is a two-stage multimodal large vision-language model training framework based on rule-based reinforcement learning, aimed at enhancing the model's general multimodal intelligence. To address the issue that existing DeepSeek R1 frameworks in multimodal scenarios mostly focus on a single task, leading to catastrophic forgetting, the project proposes a two-stage approach that first strengthens reasoning ability and then improves visual perception. Experiments show that this method can elicit metacognitive-style "Aha Moment" behavior in Qwen2.5-VL-Instruct (3B/7B), and achieves significant improvements on multiple tasks such as visual mathematics, geometric reasoning, visual counting, referring expression comprehension, visual spatial reasoning, and OCR. The project is trained based on the verl framework and has open-sourced the code, models (Ocean-R1-3B-Instruct, Ocean-R1-7B-Instruct), and datasets (stage1 63k, stage2 20k), supporting single-node and multi-node GRPO training as well as various benchmark evaluations.

Open website

About this tool

Ocean-R1 is an open and generalizable large vision-language model project that enhances multimodal intelligence through reinforcement learning. The following is a summary based on project materials of its purpose, main capabilities, usage, applicable scenarios, and limitations.

  1. Purpose Most current implementations of the DeepSeek R1 framework in multimodal environments focus on single-task specialization (such as mathematical reasoning, visual spatial analysis, referring expression comprehension REC, or visual counting). This narrow focus causes catastrophic forgetting—optimization for isolated task domains impairs the model's generalization ability through capacity competition in the shared neural substrate. In addition, existing reproduction efforts prioritize base models for stability during reinforcement learning exploration, while the key issue of metacognitive emergence ("Aha Moments") in instruction models (such as Qwen2.5-VL-Instruct) remains unresolved. Ocean-R1 aims to address the above limitations by proposing a two-stage rule-based reinforcement learning framework to enhance multimodal intelligence.

  2. Main capabilities Ocean-R1 is a two-stage rule-based reinforcement learning framework: the first stage is used to enhance the model's reasoning ability, and the second stage is used to improve visual perception. Experiments show that this method successfully elicits metacognitive emergence in Qwen2.5-VL-Instruct (3B/7B) and achieves significant improvements on multiple tasks. Specific improvement data (3B/7B):

  • Visual mathematics: MathVision (+2.7/+2.7), MathVerse (+3.2/+1.4), MathVista (+4.9/+4.4)
  • Geometric reasoning: GeoQA (+17.5/+22.2)
  • Visual counting: SuperCLEVR (+23.2/+22.6)
  • Referring expression comprehension (REC): RefCOCO/+/g average (+10.2/+1.7)
  • Visual spatial reasoning: CVBench (+9.3/+6.5)
  • OCR: OCR Bench (+9.9/+5.6) The project uses the verl framework to train models and has open-sourced all code, models, and datasets:
  • Ocean-R1-3B-Instruct
  • Ocean-R1-7B-Instruct
  • Ocean_R1_visual_data_stage1 (63k)
  • Ocean_R1_visual_data_stage2 (20k) These data come from the open-source community and have been cleaned and filtered. Project findings include:
  • Visual Aha Moment: In multimodal reasoning scenarios, rule-based reinforcement learning training on the Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct models can elicit emergent behaviors such as reflection and verification similar to an "Aha Moment".
  • Training dynamics of reasoning behaviors: On the GeoQA dataset, different models show different trends in changes to reasoning behaviors. Accuracy improves as reflective behavior increases. Once the model learns to reason, reflective behavior decreases. Multiple reflections do not necessarily produce correct results.
  • Changes in response length are related to task complexity: The 3B model improves both answer accuracy and response length through GRPO training; although the 7B model achieves similar accuracy gains, it shows an inverse relationship with response length. This indicates that model capacity is negatively correlated with verbosity, and larger models may encode information more concisely when facing tasks they consider less difficult. When RL training is applied only to the complex reasoning subset, both metrics continue to improve.
  • Cross-task knowledge transfer through compositional reasoning: After enhancing complex visual reasoning ability in the first stage, only 20k mixed samples achieve significant improvements on visual counting, referring expression comprehension, and visual spatial reasoning tasks, indicating that enhancing complex visual reasoning ability can positively transfer to simpler tasks.

III. Usage

  1. Data preparation: Training data can be downloaded from Ocean_R1_visual_data_stage1 and Ocean_R1_visual_data_stage2, and validation data can be downloaded from geoqa_test and cvbench_test. Refer to download_data.py to download the datasets and convert them to parquet format. Each data item is a dictionary, and the organization format is shown in the project materials. The prompt is: "{Question}\nYou FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE in <answer> </answer> tags."
  2. Custom reward function: Implement it in a separate fileto implement a custom reward function, and specify it via custom_reward_function.path and custom_reward_function.name. Refer to ./verl/verl/utils/reward_score/custom_reward_fn.py.
  3. Start training (GRPO): Single node:

3B

bash ./verl/examples/grpo_trainer/run_qwen25vl-3b_stage1.sh bash ./verl/examples/grpo_trainer/run_qwen25vl-3b_stage2.sh

7B

bash ./verl/examples/grpo_trainer/run_qwen25vl-7b_stage1.sh bash ./verl/examples/grpo_trainer/run_qwen25vl-7b_stage2.sh Multi-node:

3B

bash ./verl/examples/grpo_trainer/run_qwen25vl-3b_multinodes_stage1.sh bash ./verl/examples/grpo_trainer/run_qwen25vl-3b_multinodes_stage2.sh

7B

bash ./verl/examples/grpo_trainer/run_qwen25vl-7b_multinodes_stage1.sh bash ./verl/examples/grpo_trainer/run_qwen25vl-7b_multinodes_stage2.sh If you encounter bugs when running the scripts, refer to verl-install. 4. Evaluation:

  • Visual counting: SuperCLEVR
  • Geometric reasoning: GEOQA, with an example script provided for evaluation on the test set (in direct answer form).
  • Referring expression comprehension (REC): RefCOCO/+/g, you need to download and extract the COCO Train2014 images, with the image directory called <your_image_root>; download and extract the RefCOCO/+/g annotation files.
  • Visual spatial reasoning:CVBench
  • Others: OCR Bench, MathVision, MathVerse, and MathVista, evaluated using VLMEvalKit. Future directions for the project include: synthesizing more high-quality, diverse multimodal data; scaling up to larger models and more general tasks.

IV. Applicable Scenarios Suitable for research and training of multimodal large models, especially for researchers who wish to improve the general reasoning ability of vision-language models, avoid catastrophic forgetting on a single task, and explore the emergence of metacognition in instruction models; it can also be used for training and evaluation on benchmark tasks such as visual mathematics, geometric reasoning, visual counting, referring expression comprehension, visual spatial reasoning, and OCR.

V. Limitations The materials do not provide information on fees, pricing, platform, commercial use restrictions, specific hardware requirements, or training duration. The project is research-oriented; training and use require preparing data and an environment (based on the verl framework) oneself, and may involve multi-node configurations. Some benchmark evaluations rely on external resources (such as COCO images, annotation files, and VLMEvalKit).

Related tools

Browse category