SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsOther AI toolsSEED-Bench-R1

SEED-Bench-R1

SEED-Bench-R1 is a benchmark introduced by the Tencent ARC team for systematically evaluating post-training methods of multimodal large language models (MLLMs) in video understanding tasks. It contains complex real-world videos and daily planning multiple-choice questions, requiring models to possess both perception and logical reasoning abilities. The benchmark evaluates generalization through three levels: in-distribution, cross-environment, and cross-environment-task, and provides a large-scale training dataset along with easily verifiable correct answers. Using Qwen2-VL-Instruct-7B as the base model, the benchmark compares the effects of reinforcement learning (RL) and supervised fine-tuning (SFT), finding that RL is more data-efficient, performs better on in-distribution and out-of-distribution tasks, and even surpasses SFT on certain general video understanding benchmarks. The project has open-sourced the dataset, training and evaluation code, and released the training code and model checkpoints for the GRPO-CARE consistency-aware RL framework.

Open website

About this tool

SEED-Bench-R1 is a benchmark focused on video understanding and the evaluation of post-training methods for multimodal large language models, developed by the Tencent ARC team. The related paper is titled "Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1".

Purpose: SEED-Bench-R1 aims to systematically evaluate post-training methods of multimodal large language models in video understanding tasks, particularly comparing the effects of reinforcement learning (RL) and supervised fine-tuning (SFT). It focuses on complex tasks that require both perception and logical reasoning, such as daily planning.

Main capabilities:

  • Includes complex real-world videos and daily planning tasks presented in multiple-choice question form, requiring models to possess advanced perception and reasoning abilities.
  • Provides three levels of evaluation: in-distribution, cross-environment, and cross-environment-task, used to measure generalization.
  • Comes with a large-scale training dataset, and the ground-truth answers are easy to verify.
  • Uses Qwen2-VL-Instruct-7B as the base model, compares RL and SFT, demonstrates RL's advantages in data efficiency and out-of-distribution tasks, and surpasses SFT on general video understanding benchmarks such as LongVideoBench.
  • The analysis points out that RL can enhance visual perception, but often leads to reasoning chains with poorer logical coherence, and identifies key limitations such as inconsistent reasoning and ignoring visual cues.

How to use:

  • The dataset can be downloaded from HuggingFace. SEED-Bench-R1 is built on prior work EgoPlan-Bench and EgoPlan-Bench2, reusing their training and validation data. The validation data is used for Level-1 (in-distribution) and Level-2 (out-of-distribution, cross-environment) evaluation, and the test data of EgoPlan-Bench2 is used for Level-3 (out-of-distribution, cross-environment-task) evaluation.
  • The question format is multiple-choice, and the model must choose the most reasonable answer from four candidate answers, with accuracy as the main metric.
  • The training code is modified from Open-R1-Video, and the training command is configured for 8 x A100 (40GB) nodes; different hardware and topologies may require batch adjustments.Size and gradient accumulation steps.
  • Provide training scripts: grpo.sh is used to run GRPO on Qwen2-VL-7B, and sft.sh is used to run SFT.
  • For evaluation, provide the inference and evaluation results of the post-trained model.
  • The project also released the training code and corresponding model checkpoints for GRPO-CARE (a consistency-aware RL framework) (2025/06/18), as well as the SEED-Bench-R1 dataset and training/evaluation code (2025/03/31).

Applicable scenarios:

  • Research and development of multimodal large language models, especially in the field of video understanding.
  • Evaluate and compare the effects of different post-training methods (such as RL and SFT) on video understanding tasks.
  • Test the model's generalization ability in out-of-distribution scenarios.
  • Provide a reference for improving foundation model reasoning, reward modeling, and the robustness of RL to noisy signals.

Limitations:

  • The video samples come from the Epic-Kitchens and Ego4D datasets, and users must comply with the relevant license agreements; SEED-Bench-R1 does not own the copyright to these videos.
  • The training code requires a specific hardware configuration (8 x A100 40GB), and other environments may require adjustments.
  • Current analysis indicates that RL methods have limitations such as reasoning inconsistency and ignoring visual cues, which need to be improved in the future.
  • Other specific information such as features, fees, prices, platforms, effects, or usage steps is not yet provided.

Related tools

Browse category