Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding

Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, Bo Zhao · 2025

Read Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding by Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, Bo Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18478.txt.

Introduction

Watching an hour-long video is still hard for AI models, since thousands of frames swamp their memory. Xiangrui Liu and co-authors present Video-XL-Pro in this 2025 paper, a system built to compress video tokens in a way that keeps their meaning. Its core module learns, without labels, to rebuild masked video content from compact tokens.

Two parts drive the learning: a synthesizer that fakes video tokens from still image tokens, and a masking scheme that hides the most redundant pieces. A query-aware selector then finds the tokens a question needs, and a pruning strategy trims training cost. With three billion parameters, the system beat most larger rivals and handled over eight thousand frames on one graphics card.

Catalog record

CREATORXiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, Bo Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18478

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, Bo Zhao Abstract Despite advanced token compression techniques, existing multimodal large language models (MLLMs) still struggle with hour-long video understanding. In this work, we propose Video-XL-Pro, an efficient method for extremely long video understanding, built upon Reconstructive Compression of Tokens (ReCoT), a learnable module that leverages self-supervised learning to generate comprehensive and compact video tokens. ReCoT introduces two key components: (i) Dynamic Token Synthesizer (DTS): DTS generates pseudo-video tokens from static image tokens by learning intra-token relationships, which are then used in masked video modeling. (ii) Semantic-Guided Masking (SGM): SGM adaptively masks redundant visual tokens to facilitate more effective reconstructive learning. To improve training efficiency in MLLMs fine-tuning, we introduce a video-specific dataset pruning strategy and design a simple yet Query-aware Selector that enables the model to precisely locate query-relevant video tokens. With only 3B parameters, Video-XL-Pro outperforms most 7B models trained on larger datasets across multiple long video understanding benchmarks. Moreover, it can process over 8K frames on a single A100 GPU while maintaining high-quality performance. 0 0 footnotetext: \dagger Equal contribution. * Corresponding authors. Figure 1 : Compared with SOTA video understanding MLLMs, Video-XL-Pro achieves better accuracy and greater efficiency simultaneously. 1 Introduction Recently, large language models (LLMs) [ 35 , 36 , 49 ] have demonstrated remarkable capabilities in text understanding. Through visual instruction tuning [ 26 , 25 ] , which aligns visual and linguistic modalities, multimodal large language models (MLLMs) have achieved promising results in visual information reasoning. More recently, there has been a growing effort to adapt MLLMs for video understanding tasks, including video captioning and question answering [ 22 , 24 , 59 ] . However, existing methods face significant challenges when processing long videos (exceeding three minutes). Compared to single images, longer videos require sampling more frames, leading to an increased number of visual tokens, which in turn results in higher memory costs and exceeds the inherent limitations of LLMs. To address this issue, various compression modules have been proposed to reduce token counts generated by the visual encoder [ 30 , 24 , 51 , 12 ] . While these methods facilitate longer video understanding, several concerns remain. First, the quality of compressed tokens remains agnostic, as there is no established criterion to evaluate their effectiveness directly. Second, existing methods typically employ a fixed compression ratio (e.g., 8 × or 16 ×), disregarding the information intensity gap between different videos. One potential solution to this challenge is self-supervised representation learning, where Masked Autoencoders (MAE) [ 13 ] have demonstrated remarkable success by enabling a substantial portion of an image to be masked and subsequently reconstructed with high fidelity. This concept has been extended to video modeling (e.g. VideoMAE [ 48 ] ), leveraging temporal redundancy to enable efficient video reconstruction. Inspired by this, we ask: Can we develop a learnable and adaptive video compression method that enables reconstructive token compression? Ideally, if such compact video tokens can preserve the original content sufficiently for accurate reconstruction, they would greatly benefit MLLMs by enhancing both efficiency and effectiveness in video understanding. Based on this insight, we propose Video-XL-Pro, which proposes the reconstructive token compression (ReCoT) method. Specifically, ReCoT comprises two key components. \bullet Dynamic Token Synthesizer (DTS). Integrating VideoMAE with video MLLMs is a nontrivial challenge. Existing video MLLMs typically rely on image encoders such as CLIP [ 38 ] or SigLIP [ 57 ] to extract video embeddings. However, these models struggle to capture dynamic motion over time, particularly when processing high-FPS videos. While video encoders are trained to capture global video features, they are inherently limited in handling arbitrary frames, restricting their adaptability to diverse video structures. To address this limitation, we propose the Dynamic Token Synthesizer (DTS), which incorporates lightweight spatio-temporal attention blocks to enhance intra-token relationships. DTS actively learns dynamic motions within videos, enabling more effective representation learning. \bullet Semantic-guided Masking (SGM). Performing self-supervised reconstructive learning in the video domain typically requires large-scale data, leading to time-consuming training. Moreover, previous studies have primarily relied on random-based masking, which often select redundant or low-information regions, such as backgrounds, hindering effective video representation learning. To address these limitations, we propose Semantic-Guided Masking (SGM), which adaptively masks tokens by considering both intra-frame information density and inter-frame spatio-temporal significance. Instead of selecting tokens based on naive similarity calculations, SGM identifies and preserves structurally and semantically meaningful regions through a learnable mechanism, facilitating a more efficient and effective reconstruction process. By pre-training on joint image and video data, ReCoT generates comprehensive and compact tokens, facilitating MLLM fine-tuning. To mitigate the computational and time costs associated with fine-tuning for video understanding, we propose a video-specific dataset pruning strategy that selects high-quality video data based on frame-wise similarity differences. By filtering out redundant or low-informative videos, our method ensures that the model is trained on diverse, representative, and information-rich video samples, leading to improved fine-tuning efficiency and better generalization for MLLMs. Moreover, video understanding requires precise retrieval of useful information for a given query. To address this, we design a simple yet effective Query-aware selector based on retrieval-augmented generation (RAG), enabling the model to locate query-relevant video tokens for improved understanding. The effectiveness of Video-XL-Pro is validated in the following aspects. First, despite being trained on less data, the 3B Video-XL-Pro model outperforms most 7B video MLLMs across several long video understanding benchmarks, including MLVU [ 64 ] , Video-MME [ 9 ] , and LongVideoBench [ 52 ] . Second, it achieves an optimal balance between effectiveness and efficiency, processing over 8K frames on a single A100 GPU while attaining more than 95% accuracy in the Needle-in-a-Haystack evaluation. Third, the proposed ReCoT achieves comparable or superior performance to the strong SigLIP baseline on mainstream image and video benchmarks, including ImageNet [ 42 ] , UCF101 [ 47 ] , and Kinetics [ 4 ] . 2 Related work Multimodal Large Language Models. Inspired by the remarkable success of Large Language Models (LLMs), recent Multimodal Large Language Models (MLLMs) have integrated visual encoders to extract image embeddings, which are then adapted to match the LLM’s token dimension through specialized connector modules [ 17 , 2 , 21 , 65 , 26 ] . The pioneering work Kosmos-1 [ 17 ] establishes an end-to-end method that seamlessly integrates visual inputs with LLMs through a unified training paradigm. Subsequent approaches, such as Flamingo [ 2 ] and BLIP-2 [ 21 ] , utilize cross-attention layers and Q-Former modules, respectively, to combine visual and textual representations. More recent innovations, exemplified by MiniGPT-4 [ 65 ] and LLaVA [ 26 ] , further streamline the integration process by directly projecting visual features into the LLM’s embedding space via MLPs, offering more efficient paradigms for MLLMs construction. Figure 2 : Overview of Video-XL-Pro. The top part is the reconstructive token compression, in which we propose reconstructive token compression (ReCoT) to generate comprehensive and compact tokens. The bottom part is the MLLM training stage, where we propose the video dataset pruning strategy and query-aware selector to improve efficiency. Video MLLMs. Many studies have extended the capabilities of MLLMs to video understanding by encoding

Read the complete text