FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understanding

Yanan Guo, Wenhui Dong, Jun Song, Shiding Zhu, Xuan Zhang, Hanqing Yang, Yingbo Wang, Yang Du, Xianing Chen, Bo Zheng · 2025

Read FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understanding by Yanan Guo, Wenhui Dong, Jun Song, Shiding Zhu, Xuan Zhang, Hanqing Yang, Yingbo Wang, Yang Du, Xianing Chen, Bo Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20384.txt.

Introduction

Yanan Guo and co-authors present FiLA-Video, a framework that helps large vision-language models grasp long videos, in a research paper from 2025. Long videos overflow the context limits of such models, so their features must be squeezed. Past squeezing often kept redundant frames or added heavy modules.

FiLA-Video fuses several frames into one compact record with shifting weights, keeping the key content while cutting cost. A keyframe picker first chooses the most telling frames from a larger pool. The team also built a way to mint long-video training data without heavy hand labeling. Tests showed better speed and accuracy on long-video tasks than rival methods.

Catalog record

CREATORYanan Guo, Wenhui Dong, Jun Song, Shiding Zhu, Xuan Zhang, Hanqing Yang, Yingbo Wang, Yang Du, Xianing Chen, Bo Zheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20384

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understanding Yanan Guo, Wenhui Dong, Jun Song, Shiding Zhu, Xuan Zhang, Hanqing Yang, Yingbo Wang, Yang Du, Xianing Chen, Bo Zheng Abstract Recent advancements in video understanding within visual large language models (VLLMs) have led to notable progress. However, the complexity of video data and contextual processing limitations still hinder long-video comprehension. A common approach is video feature compression to reduce token input to large language models, yet many methods either fail to prioritize essential features, leading to redundant inter-frame information, or introduce computationally expensive modules.To address these issues, we propose FiLA ( Fi ne-grained Vision La nguage Model) -Video , a novel framework that leverages a lightweight dynamic-weight multi-frame fusion strategy, which adaptively integrates multiple frames into a single representation while preserving key video information and reducing computational costs. To enhance frame selection for fusion, we introduce a keyframe selection strategy, effectively identifying informative frames from a larger pool for improved summarization. Additionally, we present a simple yet effective long-video training data generation strategy, boosting model performance without extensive manual annotation. Experimental results demonstrate that FiLA-Video achieves superior efficiency and accuracy in long-video comprehension compared to existing methods. Figure 1 : Comparison between the frames selected by Uniformly Sampling and Scene-Selection respectively. We show the first 8 frames collected from the same video. The left pair shows that the method of uniformly sampling frames inevitably collects too many repeated frames, which leads to a waste of computing resources during token compression. The pair on the right solves the problem. 1 Introduction In recent years, multimodal large language models (MLLMs) such as [ 26 , 1 , 2 , 9 , 27 , 66 ] have made significant progress in the field of visual understanding. Due to the progress of these multimodal large language models in image understanding, the field of video understanding can well transfer their methods. Studies such as [ 45 ] , [ 20 ] , [ 61 ] have demonstrated the effectiveness of extending multimodal text-image large models to the video domain. However, due to the characteristics of video data that are composed of multiple image frames, how to handle an excessively long number of visual tokens has become a challenge. The existing multi-modal information processing paradigm mainly relies on LLM to process the encoded features. Therefore, the context limitations of LLM and the memory limitations restrict the processing of long videos. To address the issue, a considerable amount of research has focused on compressing video features to reduce the number of tokens ultimately submitted to the LLM. For example, VideoLLaMA2 [ 10 ] adopts the STC connector to implement spatial-temporal aggregation, preserving spatial and temporal local details while reducing the number of visual tokens. Video-ChatGPT [ 34 ] performs average-pooling on visual embeddings along spatial and temporal dimensions, respectively. LLaMA-Vid [ 23 ] utilizes text instruction to generate a contextual token and performs down-sampling on the visual embeddings of a long video as another token. However, current mainstream methods have several drawbacks. Firstly, these methods often begin by uniformly sampling a small portion of frames for processing [ 20 , 61 , 31 ] , which might overlook important frames while capturing less critical ones, leading to inadequate video understanding. Secondly, there is still room for optimization in feature processing. For example, some methods [ 41 , 54 ] introduce a module for feature merging that is overly complex, leading to a more complicated model structure, increased computation, and reduced transferability. Simple down-sampling methods [ 44 ] , commonly used in works like [ 34 , 16 ] , lack learnable characteristics. Current merging methods infrequently consider the entire video when determining which frames to merge, often damaging the event and scene information within videos, as some frames are unsuitable for merging together. Thirdly, current video understanding datasets still lack long video datasets, which is due to the lack of high-quality long-video data sources on the Internet. For example, ShareGPT4Video [ 6 ] mainly consists of 30-second videos, and llava-video-178k [ 61 ] mainly consists of videos under 3 minutes. To address these issues, we propose FiLA-Video, a novel framework for video understanding, which introduces an efficient method for spatio-temporal feature compression based on the selection of important scenes. We represent each frame with the mean of its patch tokens and use a clustering algorithm to identify the important scenes, which typically consist of a large number of similar frames. At the same time, we can obtain clustering centers as representative frames of important scenes and then select additional frames to form different scenes strictly in chronological order. Meanwhile, this approach also avoids the problem of clustering dissimilar frames together. For each scene, we focus only on temporal features and introduce a simple yet effective merging method by applying learnable weights to each patch token, greatly reducing computational complexity and simplifying the model structure, while demonstrating superior performance compared to uniform frame sampling, cross-attention, and spatio-temporal down-sampling methods. Moreover, we introduce a pipeline that utilizes high-quality short-video datasets to generate long-video captioning datasets. This method does not rely on public long videos on the Internet or complex annotation methods, yet it can still improve the model’s ability to understand long videos. Specifically, we constructed a visual captioning dataset by combining multiple short videos and their annotations. To enable the model to more accurately understand the temporal relationships within videos, we enrich the dataset with temporal cues, where video captions are segmented by scenes and correspond to specific timestamps. Experimental results show that even with the simplest uniform frame sampling method, the model can learn more knowledge from this part of the data. Overall, our contributions are as follows: • Vision-Language Model for Video Understanding : We propose FiLA-Video, a flexible framework which combines a improved clustering method to select important scenes in long videos and a simple and effective method to merge features within the scene in the spatio-temporal dimensions. Through experiments, we demonstrate its superiority over other methods. • Long Video Captioning Dataset : We introduce a long-video captioning dataset enriched with extensive timestamp information, where all data range from 5 to 30 minutes in duration. This dataset shows significant improvements on popular benchmarks with small samples. • Open-Source : We release our model checkpoints, code and dataset to the public, aiming to promote the progress of the open-source community. 2 Related Work 2.1 MLLM for Video Understanding The remarkable performance of multimodal large language models (MLLMs) has significantly advanced the field of visual understanding by effectively integrating textual and visual information. These models, such as Flamingo [ 1 ] , LLaVA [ 25 ] , extend the capabilities of traditional language models by incorporating visual context through sophisticated fusion techniques. Flamingo integrates image features with language models via a gated cross-attention mechanism, allowing the model to dynamically weigh visual and textual inputs for enhanced comprehension tasks. Approaches like LLaVA and MiniGPT4 [ 67 ] simplify the process by directly projecting visual features into the LLM embedding space through a lightweight multi-layer perceptron (MLP), bypassing complex cross-modal fusion modules. Thanks to the advancement of multi-modal large models, they can be easily transferred to tasks in the field

Read the complete text