Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1

Yi Chen , Yuying Ge , Rui Wang , Yixiao Ge , Lu Qiu , Ying Shan , Xihui Liu · 2025

Read Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1 by Yi Chen , Yuying Ge , Rui Wang , Yixiao Ge , Lu Qiu , Ying Shan , Xihui Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24376.txt.

Introduction

Yi Chen and co-authors test whether reinforcement learning beats plain fine-tuning for video AI in this 2025 paper. Their benchmark holds real-world clips with everyday planning questions in multiple-choice form. Models must both see the video well and reason over what they see.

Tests run at three levels of novelty, from familiar settings to fresh settings with fresh tasks. A mid-size open model was trained both ways. Reinforcement learning learned from less data and scored higher at every level. It even beat the fine-tuned model on outside video tests. A close read found a catch: its reasoning chains grew less coherent, and it sometimes ignored visual clues it had used well before.

Catalog record

CREATORYi Chen , Yuying Ge , Rui Wang , Yixiao Ge , Lu Qiu , Ying Shan , Xihui Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-24376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1 Yi Chen , Yuying Ge , Rui Wang , Yixiao Ge , Lu Qiu , Ying Shan , Xihui Liu Abstract Recent advancements in Chain of Thought (COT) generation have significantly improved the reasoning capabilities of Large Language Models (LLMs), with reinforcement learning (RL) emerging as an effective post-training approach. Multimodal Large Language Models (MLLMs) inherit this reasoning potential but remain underexplored in tasks requiring both perception and logical reasoning. To address this, we introduce SEED-Bench-R1, a benchmark designed to systematically evaluate post-training methods for MLLMs in video understanding. It includes intricate real-world videos and complex everyday planning tasks in the format of multiple-choice questions, requiring sophisticated perception and reasoning. SEED-Bench-R1 assesses generalization through a three-level hierarchy: in-distribution, cross-environment, and cross-environment-task scenarios, equipped with a large-scale training dataset with easily verifiable ground-truth answers. Using Qwen2-VL-Instruct-7B as a base model, we compare RL with supervised fine-tuning (SFT), demonstrating RL’s data efficiency and superior performance on both in-distribution and out-of-distribution tasks, even outperforming SFT on general video understanding benchmarks like LongVideoBench. Our detailed analysis reveals that RL enhances visual perception but often produces less logically coherent reasoning chains. We identify key limitations such as inconsistent reasoning and overlooked visual cues, and suggest future improvements in base model reasoning, reward modeling, and RL robustness against noisy signals. † † footnotetext: Corresponding Authors. Figure 1: (a) Overview of SEED-Bench-R1 (SBR), which systematically evaluates post-training methods for MLLMs in video understanding. SBR features a three-level evaluation hierarchy: in-distribution, cross-environment, and cross-environment-task scenarios, equipped with training data containing easily verifiable ground-truth answers, to assess generalization across different levels . These tasks necessitate both perception and reasoning to tackle complex real-world challenges. (b) Notably, an MLLM trained using reinforcement learning via GRPO outperforms both the base model and supervised fine-tuning (SFT) model, particularly in out-of-distribution scenarios (e.g., levels 2–3). Additionally, this RL-trained model exhibits strong generalization capabilities across general video understanding benchmarks (e.g., LongVideoBench). 1 Introduction Recent advancements in the reasoning capabilities of Large Language Models (LLMs) ( Guo et al. 2025 ; ope 2024 ; Team et al. 2025 ) have been driven by progress in long Chain of Thought (COT) generation. Among the various approaches to enhancing long COT, reinforcement learning (RL) ( Shao et al. 2024 ; Ouyang et al. 2022 ; Yeo et al. 2025 ) has emerged as a particularly effective post-training method. RL enables LLMs to refine their COT through self-evolution with verifiable rewards, resulting in models that excel at solving complex problems and demonstrate strong generalization in out-of-distribution (OOD) tasks. Multimodal Large Language Models (MLLMs), which are built upon LLMs, inherit their reasoning potential while incorporating additional modules for processing multimodal inputs. This has led to a growing interest in whether RL can similarly enhance multimodal understanding in MLLMs ( Zhang et al. 2025 ; Liu et al. 2025 ; Meng et al. 2025 ) . However, existing research primarily focuses on image-based tasks, emphasizing either perception (e.g., detection and grounding) or logical inference (e.g., multimodal math problem solving). We argue that an ideal testbed for studying the impact of post-training methods on multimodal understanding should balance both perception and logical inference , ensuring models can integrate these capabilities to derive accurate answers. Additionally, the generalization capability of MLLMs must be rigorously evaluated to assess the robustness of post-training methods. To address this, we introduce SEED-Bench-R1 , a benchmark designed for systematic evaluation of the effectiveness of post-training methods on video understanding. As shown in Table 1 , SEED-Bench-R1 is built on our prior works, reusing the training and validation data from our EgoPlan-Bench ( Chen et al. 2023 ) , as well as the test data from our EgoPlan-Bench2 ( Chen et al. 2023 ) . Generally, the benchmark leverages realistic egocentric videos ( Damen et al. 2022 ; Grauman et al. 2022 ) capturing everyday human activities, requiring models to understand open-form task goals, track long-horizon visual progress, perceive complex environmental observations, and reason about next actions using world knowledge. Specifically, the validation data from EgoPlan-Bench are used for Level-1 (in-distribution) and Level-2 (OOD, cross-environment) evaluation, while the test data from EgoPlan-Bench2 cover more general domains and are used for Level-3 (OOD, cross-environment-task) evaluation. As shown in Figure 1 , SEED-Bench-R1 features: 1) intricate real-world visual inputs necessitating nuanced perception , 2) diverse questions requiring logical inference with common sense , 3) rigorously partitioned validation sets to assess in-distribution (L1) and OOD (L2/L3) generalization capabilities , and 4) large-scale, automatically constructed training questions with easily verifiable ground-truth answers (the ground-truth answer comes from the actual next action occurring right after the current observation in the original uncropped video). We evaluate representative post-training methods on SEED-Bench-R1 using Qwen2-VL-Instruct-7B ( Wang et al. 2024 ) as the base model, comparing RL (specifically GRPO ( Shao et al. 2024 ) ) with supervised fine-tuning (SFT). Our experiments reveal that RL is highly data-efficient, significantly improving performance on both in-distribution (L1) and OOD (L2/L3) questions, even with simple outcome-based rewards. RL’s superiority over SFT is particularly pronounced in OOD scenarios and extends to general video understanding benchmarks like LongVideoBench ( Wu et al. 2024 ) . Our analysis further explores how RL influences COT generation and its impact on visual perception and logical inference. We find that RL enhances the model’s ability to attend to visual content more effectively, particularly in OOD scenarios. RL teaches the model to dynamically query visual inputs with COT tokens rather than memorizing superficial reasoning patterns, thus achieving better generalization performance. However, limitations remain, such as the model’s occasional neglect of key visual cues due to limited perception granularity and the lack of logical coherence in generated reasoning chains without process supervision, which hinder transparency and performance. Split # Samples Domain Cross-Environment Cross-Task Video Source Benchmark Source Train 50,269 Daily life - - Epic-Kitchens EgoPlan-Bench Val-L1 2,432 Daily life \times \times Epic-Kitchens EgoPlan-Bench Val-L2 923 Daily life \surd \times Ego4D EgoPlan-Bench Val-L3 1,321 Hobbies, Daily life, Recreation, Work \surd \surd Ego4D EgoPlan-Bench2 Table 1: Data statistics of SEED-Bench-R1, which consists of a training set and a hierarchical three-level validation set for in-distribution, cross-environment, and cross-environment-task evaluations. Figure 2: Example questions from the three-level evaluation hierarchy in SEED-Bench-R1’s validation set, including in-distribution, cross-environment, and cross-environment-task scenarios. We conclude by outlining promising directions for future work, including eliciting stronger base reasoning abilities before RL, improving reward modeling to balance visual perception and logical inference, and enhancing RL’s robustness to noisy reward signals. These steps are crucial for scaling RL to larger datasets and achieving more reliable alignment in multimodal understanding. In summary, our contributions are as below: • We introduce SEED-Bench-R1, a video understanding benchmark designed to evaluate post-training methods, featuring real-world egocentric videos, questions balancing perception and logical inference, and a large-scale training dataset with rigorous validation splits divided into a three-level hierarchy for generalization assessment. • Using SEED-Bench-R1, we conduct a systematic study and demonstrate that reinforcement learning (RL)—specifically GRPO—outperforms supervised fine-tuning (SFT) in data efficiency and generalization, notably in OOD scenarios, even with simple outcome-based rewards. • Our analysis reveals that RL enhances video understanding by improving visual attention and encouraging dynamic querying of

Read the complete text