Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Sanghyeok Chu, Seonguk Seo †, Bohyung Han · 2025
Read Fine-Grained Captioning of Long Videos through Scene Graph Consolidation by Sanghyeok Chu, Seonguk Seo †, Bohyung Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16427.txt.
Introduction
Sanghyeok Chu, Seonguk Seo, and Bohyung Han caption long videos in fine detail in a 2025 paper. Image captioners see only moments; strung across an hour, their words lose the plot. Retraining giant models on long video is dear, and older merge tricks lose detail.
Their method captions each short segment with a stock model, then parses those captions into scene graphs: who did what to whom, where. The graphs merge into one big graph that keeps both the story arc and the small facts. A light decoder turns the merged graph back into flowing text. The result beats language-model mergers in tests, with no long-video training and far less compute.
Catalog record
| CREATOR | Sanghyeok Chu, Seonguk Seo †, Bohyung Han |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16427
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fine-Grained Captioning of Long Videos through Scene Graph Consolidation Sanghyeok Chu, Seonguk Seo †, Bohyung Han Abstract Recent advances in vision-language models have led to impressive progress in caption generation for images and short video clips. However, these models remain constrained by their limited temporal receptive fields, making it difficult to produce coherent and comprehensive captions for long videos. While several methods have been proposed to aggregate information across video segments, they often rely on supervised fine-tuning or incur significant computational overhead. To address these challenges, we introduce a novel framework for long video captioning based on graph consolidation. Our approach first generates segment-level captions, corresponding to individual frames or short video intervals, using off-the-shelf visual captioning models. These captions are then parsed into individual scene graphs, which are subsequently consolidated into a unified graph representation that preserves both holistic context and fine-grained details throughout the video. A lightweight graph-to-text decoder then produces the final video-level caption. This framework effectively extends the temporal understanding capabilities of existing models without requiring any additional fine-tuning on long video datasets. Experimental results show that our method significantly outperforms existing LLM-based consolidation approaches, achieving strong zero-shot performance while substantially reducing computational costs. 1 Introduction Vision-language models (VLMs) have demonstrated impressive capabilities across diverse vision-language tasks, including visual question answering, visual dialogue, cross-modal retrieval, and spatiotemporal understanding ( Alayrac et al., 2022 ; Dai et al., 2023 ; OpenAI, 2023 ; Chen et al., 2024b ; Huang et al., 2024 ; Zhang et al., 2025 ; Xu et al., 2024 ; Maaz et al., 2024 ) . Notably, substantial progress has been made in generating captions for images and short video clips ( Liu et al., 2024 ; Chai et al., 2025 ; Zhao et al., 2024 ; Wang et al., 2024 ; Chen et al., 2024a ; Mun et al., 2019 ) . However, generating captions for longer videos remains a significant challenge. Most existing models are designed for short-term visual inputs, such as images or short video clips, and lack effective support for holistic encoding of entire long videos. As a result, captioning videos beyond a model’s temporal window typically requires processing and integrating information from multiple temporal segments. Several approaches, such as memory-based ( Zhou et al., 2024 ; Song et al., 2024 ; Balazevic et al., 2024 ) and recursive frameworks ( Zhou et al., 2024 ; Islam et al., 2024 ; Qian et al., 2024 ; Weng et al., 2024 ; Kahatapitiya et al., 2024 ) , have been proposed to consolidate information across these segments. However, these methods often rely on supervised fine-tuning with the target datasets, which limits their generalizability to unseen video domains. More recently, large language models (LLMs) have been employed to generate textual summaries across multiple video segments ( Wang et al., 2022b ; Chen et al., 2023 ; Zhang et al., 2024a ) . While these LLM-based approaches eliminate the need to adapt existing models for long videos, they typically incur high inference overhead and require significant computational resources. To address these limitations, we propose a novel framework that integrates segment-level captions into a unified global description via graph-based consolidation. We first obtain segment-level captions—each corresponding to either a single frame or a short video clip, depending on the chosen visual captioning model—using an off-the-shelf captioning algorithm. Each caption is then parsed into a scene graph, and these graphs are consolidated into a unified structure that captures the comprehensive semantics of the entire video. Finally, a lightweight graph-to-text decoder, trained solely on external text corpora, translates the consolidated graph into a coherent global caption. The proposed approach enhances understanding and processing of long-range temporal information without requiring architectural changes or fine-tuning on long video datasets. In particular, our framework can be paired with any off-the-shelf VLM, effectively extending its captioning capability beyond the model’s inherent temporal constraints. Unlike other LLM-based consolidation methods, it minimizes computational overhead by employing a lightweight graph-to-text decoder with significantly fewer parameters. Our experimental results demonstrate that our approach achieves superior performance in both zero-shot video captioning and zero-shot video paragraph captioning, demonstrating its effectiveness and efficiency. In summary, our key contributions are organized as follows: • We propose a novel approach to generate fine-grained captions for long videos using the information across multiple temporal segments. • We introduce a graph consolidation algorithm that merges segment-level scene graphs into a unified representation to capture both holistic context and fine-grained details across the entire video. • Our method achieves strong zero-shot captioning performance with significantly lower computational cost compared to LLM-based approaches. 2 Related Works Video captioning Recent advances in video captioning have predominantly rely on supervised training using large-scale datasets, achieving impressive results across various benchmarks ( Lei et al., 2021 ; Wang et al., 2022a ; Yan et al., 2022 ; Liu et al., 2024 ; Zhao et al., 2024 ; Wang et al., 2024 ; Chen et al., 2024a ) . However, extending these supervised approaches to longer videos remains challenging, primarily due to the scarcity of annotated data covering extensive temporal contexts and the computational complexity involved in modeling long-range dependencies. While various methods have been proposed to tackle these challenges, the needs for supervised fine-tuning for specific target datasets hampers scalability and generalization to unseen video domains ( Yang et al., 2023 ; Islam et al., 2024 ; Song et al., 2024 ; Balazevic et al., 2024 ; Qian et al., 2024 ; Weng et al., 2024 ; Kahatapitiya et al., 2024 ) . Zero-shot video captioning Researchers have explored methods for video captioning without using paired video-text annotations. One approach involves refining language model outputs solely at test time. ZeroCap ( Tewel et al., 2022 ) and related methods ( Tewel et al., 2023 ) use image-text alignment score calculated by CLIP ( Radford et al., 2021 ) in gradient updates to adjust language model features, while MAGIC ( Su et al., 2022 ) employs a CLIP-induced decoding strategy to ensure semantic relevance. Although initially developed for images, these methods extend to videos by aggregating frame-level features into a single representation. Another approach, often termed zero-shot, involves text-only training without paired video-text annotations, where text decoders are used in conjunction with image-text aligned encoders such as CLIP and ImageBind ( Girdhar et al., 2023 ) . Methods such as DeCap ( Li et al., 2023b ) and C 3 ( Zhang et al., 2024b ) generate captions by aligning visual and textual features in a shared embedding space. However, these approaches often fail to produce accurate and coherent captions, especially when applied to videos with complex events. Zero-shot long video captioning Generating coherent and comprehensive captions for long-context videos under zero-shot settings often relies on the consolidation of information derived from multiple temporal segments. Existing consolidation techniques, including memory-based ( Zhou et al., 2024 ; Song et al., 2024 ; Balazevic et al., 2024 ) and recursive approaches ( Islam et al., 2024 ; Qian et al., 2024 ; Weng et al., 2024 ; Kahatapitiya et al., 2024 ) , require supervised fine-tuning on the target dataset, which limits their applicability to zero-shot scenarios. Recently, LLMs have emerged as a promising tool for zero-shot consolidation,