VideoSAVi: Self-Aligned Video Language Models without Human Supervision

Yogesh Kulkarni, Pooyan Fazli · 2024

Read VideoSAVi: Self-Aligned Video Language Models without Human Supervision by Yogesh Kulkarni, Pooyan Fazli (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00624.txt.

Introduction

Yogesh Kulkarni and Pooyan Fazli teach video language models to improve themselves without human labels or paid model judges. Standard tuning needs ranked pairs of answers, which are costly to collect for video. Their pipeline, VideoSAVi, has the model critique its own first answers, spot reasoning errors, and write better versions. Those before-and-after pairs then train the model through preference optimization, round after round.

The self-trained model gained 4.2 points on MVBench, 3.9 on PerceptionTest, and 6.8 on the hard EgoSchema benchmark over its starting point. It needs only thirty-two video frames per question, keeping compute modest. The method works with different base models, since nothing in the loop depends on one architecture. The authors present self-critique as a cheap route to sharper video reasoning where labeled data is scarce.

Catalog record

CREATORYogesh Kulkarni, Pooyan Fazli
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00624

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni, Pooyan Fazli Abstract Recent advances in video-large language models (Video-LLMs) have led to significant progress in video understanding. Current preference optimization methods often rely on proprietary APIs or ground-truth captions to generate preference data (i.e., pairs of model outputs ranked based on their quality or alignment with human judgment), which is then used to train models for video-language alignment. This approach is both costly and labor-intensive. To address this limitation, we introduce VideoSAVi ( S elf- A ligned Vi deo Language Model), a self-training pipeline that enables Video-LLMs to reason over video content without external supervision. Our approach includes a self-critiquing mechanism that identifies reasoning errors in the model’s initial responses and generates improved alternatives, creating preference pairs directly from video content. VideoSAVi then applies Direct Preference Optimization (DPO), which uses the preference data to iteratively train the model, enhancing temporal and spatial reasoning in video understanding. Experiments show that VideoSAVi achieves state-of-the-art performance on MVBench (74.0%) and delivers significant improvements across other benchmarks, including a 3.9% gain on PerceptionTest and a substantial 6.8% improvement on the challenging EgoSchema dataset compared to baseline models. Our model-agnostic approach is computationally efficient, requiring only 32 frames, offering a promising direction for self-aligned video understanding without reliance on external models or annotations. 1 Introduction Vision-language models (VLMs) ( Liu et al. 2023 ; Li et al. 2023 ; Radford et al. 2021 ; Wang et al. 2024b ; Chen et al. 2024 ; Li et al. 2024a ) have made significant strides by integrating visual perception with the reasoning capabilities of large language models (LLMs) ( OpenAI 2024 ; Dubey et al. 2024 ; Ouyang et al. 2022 ) . These models excel in interpreting and generating contextually relevant responses through the combination of image encoders and language generation techniques. Building on this foundation, recent video-large language models (Video-LLMs) ( Zhang et al. 2023 ; Lin et al. 2024 ; Li et al. 2024b ; Zhang et al. 2024d ; Wang et al. 2022 ; Wang et al. 2024e ) incorporate temporal dimensions, enabling comprehensive video understanding by transforming video frames into tokens that LLMs can process. While Video-LLMs demonstrate impressive capabilities, they typically require vast, high-quality annotated datasets, making them resource-intensive and limiting their scalability. Instruction tuning has been pivotal in advancing both VLMs and Video-LLMs ( Liu et al. 2023 ; Brown et al. 2020 ; Xu et al. 2023 ; Wei et al. 2021 ; Wang et al. 2024b ; Chen et al. 2024 ; Zhang et al. 2024c ) , but creating extensive video instruction datasets incurs substantial costs ( Deng et al. 2024 ) . Recent efforts to generate large instruction datasets (up to 1.3M pairs) by distilling knowledge from proprietary models like GPT-4V ( Zhang et al. 2024d ) have yielded only marginal improvements. This dependence on extensive annotated data and proprietary models restricts the adaptability of Video-LLMs, posing a barrier to broader applications. Alignment-based approaches have shown promise for improving video understanding. LLaVA-Hound ( Zhang et al. 2024b ) uses Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) with text-based ranking of preference data generated by proprietary models, while TPO ( Li et al. 2025 ) targets temporal preference optimization by sampling negative examples from video segments not present in the target clip. However, these methods depend heavily on expensive proprietary APIs or ground truth captions, limiting their accessibility and scalability. This raises a critical research question: How can we train Video-LLMs to generate high-quality outputs without relying on proprietary models, ground-truth captions, or costly human annotations while maintaining robust temporal and spatial reasoning? To address this challenge, we propose VideoSAVi, a novel framework that requires no external supervision beyond the model itself. Our approach focuses on post-training refinement through a four-stage process: (1) generating diverse reasoning questions and initial answers about video content, (2) self-critiquing these answers to identify factual errors and reasoning flaws, (3) revising the responses based on the self-generated feedback, and (4) leveraging these pairs for DPO to align the model towards improved reasoning capabilities. Unlike prior approaches that depend on proprietary models or extensive annotations, VideoSAVi uniquely generates high-quality preference pairs through self-critique—identifying specific reasoning failures in both spatial relationships and temporal sequences. This self-supervision creates subtle yet unambiguous learning signals that, when optimized through DPO, enable efficient post-training refinement of capabilities already present in the model. By directly addressing the model’s own weaknesses rather than conforming to external judgment, our approach delivers significant improvements across diverse benchmarks without the computational cost of full retraining. Our main contributions are as follows: 1. We introduce VideoSAVi, a novel self-training framework that enables Video-LLMs to reason over video content, eliminating the need for external supervision while maintaining computational efficiency. 2. We develop a quality-aware self-critique mechanism that generates preference pairs, focusing on both temporal and spatial reasoning, enabling comprehensive video understanding without external supervision. 3. Through extensive experiments, we demonstrate that VideoSAVi achieves state-of-the-art performance on MVBench (74.0%) and delivers significant improvements across multiple benchmarks, including +3.6% on NeXTQA, +3.9% on PerceptionTest, and +6.8% on EgoSchema. Moreover, VideoSAVi shows consistent effectiveness across various model architectures, with an average improvement of +3.4%. 2 Related Work Video-LLMs. Recent Video-LLMs have made remarkable progress in multimodal understanding capabilities ( Chen et al. 2024 ; Wang et al. 2024b ; Li et al. 2024a ; Li et al. 2024b ; Zhang et al. 2024d ) . However, these models typically require massive instruction-tuning datasets and extensive computational resources for pre-training. Even with substantial training data, they often struggle with proper grounding in visual content ( Fu et al. 2024 ; Zhou et al. 2024 ; Liu et al. 2024b ) , particularly for videos ( Wang et al. 2024a ; Yao et al. 2024 ; Wang et al. 2024e ; Song et al. 2024 ) . While some approaches attempt to address these challenges through architecture modifications ( Liu et al. 2024a ; Shu et al. 2024 ; Li et al. 2024d ; Wang et al. 2024d ) , specialized training objectives ( Lee et al. 2024 ; Lan et al. 2024 ; Liu et al. 2024c ) , or inference-time adaptations ( Yang et al. 2024 ; Hu et al. 2025 ; Xu et al. 2024 ) , they typically focus on specific aspects rather than holistic alignment. In contrast to these approaches that emphasize training from scratch or adding specialized components, VideoSAVi offers a post-training refinement strategy. Rather than attempting to teach new capabilities, we focus on identifying, amplifying, and improving specific reasoning skills already present in the model. This targeted enhancement allows us to efficiently boost performance in particular domains without the computational costs of full retraining. By leveraging self-alignment through preference optimization, VideoSAVi helps models rectify their own errors and strengthen existing capabilities across temporal and spatial dimensions without depending on external supervision or extensive new datasets. Learning from AI Feedback. Recent work has adapted preference learning techniques to video-language models to improve alignment. LLaVA-Hound-DPO ( Zhang et al. 2024b ) introduced using Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) for video understanding, but operates primarily at the

Read the complete text