Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization

Pritam Sarkar Queen’s University, Canada and Vector Institute [email protected] &Ali Etemad Queen’s University, Canada [email protected] Website Code Models · 2025

Read Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization by Pritam Sarkar Queen’s University, Canada and Vector Institute [email protected] &Ali Etemad Queen’s University, Canada [email protected] Website Code Models (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12083.txt.

Introduction

Pritam Sarkar and Ali Etemad proposed a self-alignment method in 2025 for large video language models. Such models still miss fine timing detail, invent content, and err on simple video questions. Those faults block safe use in the real world.

Their frame builds training pairs from the model's own common errors, such as weak time sense or over-trust in language cues. A new tuning method then uses refined rewards at the sub-sequence level and per-token limits on drift. That gave more precise alignment and steadier training than a standard method. Tests covered hallucination, short and long video understanding, and fine timing tasks.

Catalog record

CREATORPritam Sarkar Queen’s University, Canada and Vector Institute [email protected] &Ali Etemad Queen’s University, Canada [email protected] Website Code Models
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12083

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization Pritam Sarkar Queen’s University, Canada and Vector Institute [email protected] &Ali Etemad Queen’s University, Canada [email protected] Website Code Models Abstract Despite recent advances in Large Video Language Models (LVLMs), they still struggle with fine-grained temporal understanding, hallucinate, and often make simple mistakes on even simple video question-answering tasks, all of which pose significant challenges to their safe and reliable deployment in real-world applications. To address these limitations, we propose a self-alignment framework that enables LVLMs to learn from their own errors. Our proposed framework first obtains a training set of preferred and non-preferred response pairs, where non-preferred responses are generated by incorporating common error patterns that often occur due to inadequate spatio-temporal understanding, spurious correlations between co-occurring concepts, and over-reliance on linguistic cues while neglecting the vision modality, among others. To facilitate self-alignment of LVLMs with the constructed preferred and non-preferred response pairs, we introduce Refined Regularized Preference Optimization (RRPO), a novel preference optimization method that utilizes sub-sequence-level refined rewards and token-wise KL regularization to address the limitations of Direct Preference Optimization (DPO). We demonstrate that RRPO achieves more precise alignment and more stable training compared to DPO. Our experiments and analysis validate the effectiveness of our approach across diverse video tasks, including video hallucination, short- and long-video understanding, and fine-grained temporal reasoning. 1 Introduction Figure 1 : Examples of failure cases in simple video understanding tasks by state-of-the-art LVLMs, and improvements observed after self-alignment with RRPO. Despite recent progress in Large Video Language Models (LVLMs) [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] , these models continue to face limitations in fine-grained temporal understanding [ 16 , 17 , 18 ] , demonstrate a propensity for hallucination [ 19 , 20 ] , struggle with long-video understanding [ 21 , 22 , 23 , 24 , 25 ] , and frequently make naive mistakes in short-video question-answering tasks [ 16 , 17 , 18 ] . Please see Figure 1 for a number of examples. These shortcomings severely limit their safe and reliable deployment in real-world applications. The underlying causes of these limitations are multifaceted, encompassing factors such as inadequate spatial and temporal understanding [ 16 , 17 , 18 ] , vision-language representational misalignments [ 26 , 22 ] , challenges in processing long visual sequences due to context length constraints [ 21 , 22 , 23 ] , spurious correlations between co-occurring concepts [ 27 , 28 ] , and an over-reliance on linguistic cues while neglecting the visual information [ 16 , 17 , 18 , 29 ] . Figure 2 : An overview of our self-alignment framework. To address these shortcomings and enhance video understanding in LVLMs, we design a self-alignment [ 30 ] framework that enables LVLMs to learn from their own errors. Specifically, we begin by sampling video-question pairs from an open-source benchmark. We then apply spatio-temporal perturbations to the video content to mimic common errors that often arise from over-reliance on linguistic cues, spurious correlations between co-occurring concepts, and insufficient spatio-temporal understanding. Using the perturbed video and the corresponding question, we perform inference with the target LVLM. If the model’s response is incorrect, we construct a self-alignment pair by treating the incorrect response as a non-preferred sample and the correct response as the preferred one, which is kept for self-alignment training. Responses that are already correct are discarded, as they offer limited self-improvement potential. Subsequently, we optimize a loss function to prioritize the preferred response over the non-preferred one. Notably, our data generation pipeline is free from human annotation and can be easily scaled. A high-level overview of our self-alignment framework is depicted in Figure 2 . To effectively train LVLMs using the generated preferred and non-preferred response pairs, we introduce Refined Regularized Preference Optimization (RRPO) , an approach designed to address limitations of Direct Preference Optimization (DPO) [ 31 ] . RRPO is designed to address the key drawbacks of DPO. RRPO provides a fine-grained sub-sequence-level reward to explicitly penalize the tokens containing the key concepts that are different between the preferred and non-preferred response pairs. This is contrary to DPO’s response-level reward which penalizes all tokens within non-preferred responses, thus lacking precision and often proving unsuitable for fine-grained alignment. It should be noted that our fine-grained reward further benefits from computing a smaller gradient during optimization and is therefore less prone to diverge away from its initial state unlike DPO’s response-level feedback along with weak regularization which can cause significant divergence from the base model, leading to suboptimal performance. However, sub-sequence-level rewards may incentivize the model to exploit shortcuts, such as outputting correct concepts without proper context or complete responses. To mitigate this, we also minimize token-wise KL-divergence [ 27 ] using a reference model on the preferred responses. This ensures that the LVLM maintains a tight bound on its likelihood across the entirety of the preferred response while reducing the likelihood on the non-preferred concepts. Through empirical and theoretical analysis, we demonstrate that RRPO exhibits greater stability and smoother convergence during optimization compared to DPO. We validate our method on three popular LVLMs specialized for video understanding, VideoChat2, LLaVA-Video, and longVU, covering a wide range of architectures, LLMs, vision encoders, and training setups. Our in-depth evaluation demonstrates that our proposed self-alignment reduces hallucinations and improves performance in fine-grained temporal understanding, as well as in both short- and long-video understanding tasks. In summary, our contributions are as follows: • We design a self-alignment framework to facilitate self-improvement of LVLMs based on their own errors. We introduce RRPO , a preference optimization method that addresses the limitations of DPO by utilizing sub-sequence-level refined rewards and token-wise strong KL regularizer, resulting in more precise alignment and stable training. • Our rigorous evaluation demonstrates the effectiveness of our proposed method across diverse video tasks, including video hallucination, short and long video understanding, and fine-grained temporal reasoning, among others. Moreover, our experimental and theoretical analyses highlight the superiority of RRPO over DPO in aligning LVLMs. We make our code, data, and model weights public to enable fast and accurate reproducibility. 2 Preference Responses for Self-alignment Figure 3 : An example of perturbed video. Figure 4 : A few training samples. We bold the correct and incorrect concepts. As the first step in our framework, we construct a training dataset \mathcal{D} comprising both preferred responses and responses containing incorrect concepts in order to align the LVLM \pi_{\theta} to favor correct concepts over incorrect ones. We begin by utilizing a large and diverse publicly available video instruction tuning dataset, containing triplets of video x_{v} , question x_{q} , and their human-preferred answers y^{+} . To generate non-preferred responses y^{-} , we obtain perturbed videos \hat{x}_{v}\!=\!f_{p}(x_{v}) , where f_{p} is a perturbation function which masks a large portion of frames and applies temporal shuffling, compromising spatio-temporal consistency. Our intention from this step is to provoke the LVLM to generate responses primarily based on language cues or their parametric knowledge. These perturbed videos \hat{x}_{v} are fed to \pi_{\theta} as inputs to generate responses with potential

Read the complete text