Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

Minghao Qin 1∗ Xiangrui Liu 1,2∗ Zhengyang Liang 1∗ Yan Shu 1,3 Huaying Yuan 1,4 Juenjie Zhou 1,5 Shitao Xiao 1 Bo Zhao 1,2 Zheng Liu 1,6 1 Beijing Academy of Artificial Intelligence 2 Shanghai Jiao Tong University 3 University of Trento 4 Renmin University of China 5 Beijing University of Posts and · 2025

Read Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification by Minghao Qin 1∗ Xiangrui Liu 1,2∗ Zhengyang Liang 1∗ Yan Shu 1,3 Huaying Yuan 1,4 Juenjie Zhou 1,5 Shitao Xiao 1 Bo Zhao 1,2 Zheng Liu 1,6 1 Beijing Academy of Artificial Intelligence 2 Shanghai Jiao Tong University 3 University of Trento 4 Renmin University of China 5 Beijing University of Posts and (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19225.txt.

Introduction

Minghao Qin and co-authors present a video AI paper, published in 2025, on understanding very long videos without exhausting computer memory. Language models that watch video must store keys and values for every frame they attend to, and that store grows huge over thousands of frames. Their system, Video-XL-2, trims that store based on what the task at hand actually needs.

Incoming frames are processed in chunks, with full attention inside each chunk and sparse attention across them. During answer generation, the system reloads detailed or sparse records per chunk depending on relevance. The method topped open lightweight rivals on long video benchmarks and handled more than ten thousand frames at once, pairing accuracy with striking efficiency.

Catalog record

CREATORMinghao Qin 1∗ Xiangrui Liu 1,2∗ Zhengyang Liang 1∗ Yan Shu 1,3 Huaying Yuan 1,4 Juenjie Zhou 1,5 Shitao Xiao 1 Bo Zhao 1,2 Zheng Liu 1,6 1 Beijing Academy of Artificial Intelligence 2 Shanghai Jiao Tong University 3 University of Trento 4 Renmin University of China 5 Beijing University of Posts and
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19225

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification Minghao Qin 1∗ Xiangrui Liu 1,2∗ Zhengyang Liang 1∗ Yan Shu 1,3 Huaying Yuan 1,4 Juenjie Zhou 1,5 Shitao Xiao 1 Bo Zhao 1,2 Zheng Liu 1,6 1 Beijing Academy of Artificial Intelligence 2 Shanghai Jiao Tong University 3 University of Trento 4 Renmin University of China 5 Beijing University of Posts and Telecommunications 6 Hong Kong Polytechnic University Homepage: https://unabletousegit.github.io/video-xl2.github.io/ Abstract Multi-modal large language models (MLLMs) models have made significant progress in video understanding over the past few years. However, processing long video inputs remains a major challenge due to high memory and computational costs. This makes it difficult for current models to achieve both strong performance and high efficiency in long video understanding. To address this challenge, we propose Video-XL-2, a novel MLLM that delivers superior cost-effectiveness for long-video understanding based on task-aware KV sparsification . The proposed framework operates with two key steps: chunk-based pre-filling and bi-level key-value decoding . Chunk-based pre-filling divides the visual token sequence into chunks, applying full attention within each chunk and sparse attention across chunks. This significantly reduces computational and memory overhead. During decoding, bi-level key-value decoding selectively reloads either dense or sparse key-values for each chunk based on its relevance to the task. This approach further improves memory efficiency and enhances the model’s ability to capture fine-grained information. Video-XL-2 achieves state-of-the-art performance on various long video understanding benchmarks, outperforming existing open-source lightweight models. It also demonstrates exceptional efficiency, capable of processing over 10,000 frames on a single NVIDIA A100 (80GB) GPU and thousands of frames in just a few seconds. Video-XL-2 has been made publicly available at this repo . 1 1 footnotetext: Equal contribution. 1 Introduction Multi-modal large language models (MLLMs) have demonstrated remarkable progress in visual tasks that demand perception and reasoning abilities reid2024gemini ; gpt4o ; Claude3 ; bai2025qwen2 . Recently, more and more research has extended MLLMs to video understanding zhang2025videollama ; li2024videochat ; liu2025nvila ; li2024llava ; chen2025eagle . given their significant potential in modeling temporal visual data. Existing MLLMs struggle with a major challenge in video understanding: the prohibitively high computational and memory costs, especially for long videos. Specifically, long videos consist of numerous frames, each producing a huge amount of visual tokens after visual encoded. As a result, the input often exceeds the model’s context window, making it computationally expensive and impractical for real-world applications. Even with extended context lengths via modifying model architecture or training strategy, processing such a large number of visual tokens remains inefficient and resource-intensive wang2024longllava ; longva ; chen2025eagle . The primary solution to this problem is token reduction. Existing methods typically compress or downsample visual tokens per frame via an additional module llama-vid2023 ; videollama ; li2023videochat ; li2024videochat ; moviechat2023 . While these approaches certainly alleviate some memory and computational demands, they fundamentally struggle with the quadratic growth of computational FLOPs relative to the total number of input tokens. Consequently, for increasingly long video inputs, these models still face immense resource burdens and often incur critical information loss. Therefore, we propose Video-XL-2, a model featuring both advanced video understanding capabilities and outstanding efficiency, specifically designed for long video understanding. \bullet Advanced Performance Construction : Video-XL-2’s advanced video understanding capabilities are fundamentally built upon two pillars: its innovative model architecture and an incremental training strategy. For the architecture, we adopt and refine the foundational structure of Video-XL-Pro liu2025videoxlproreconstructivetokencompression . Its main characteristic is an innovative module, Dynamic Token Synthesize (DTS), capable of effectively preserving rich visual information while mitigating redundancy across adjacent frames. Meanwhile, our incremental training strategy is designed to progressively robustly and comprehensively construct Video-XL-2’s visual understanding abilities. This phased approach gradually equips the model with strong visual comprehension skills, ensuring its superior performance across diverse long video understanding tasks. \bullet Comprehensive Efficiency Optimization : Beyond its powerful understanding capabilities, Video-XL-2 integrates a comprehensive efficiency optimization strategy based on task-aware KV sparsification. Large Language Model (LLM) inference consists of two stages: pre-filling and decoding. In the pre-filling stage, Key-Value (KV) pairs corresponding to all input tokens are generated and stored in the KV cache. The decoding stage then iteratively generates output tokens one by one. Previous work jiang2407minference ; li2024snapkv ; xiao2023efficient ; zhang2023h2o ; yuan2025native ; lu2025moba has observed significant KV sparsity in both pre-filling and decoding. Based on this observation, we propose two main innovations, Chunk-based Pre-filling and Bi-level Key-Values (KVs) Decoding, to significantly alleviate the computational and memory burdens in both stages for LVU. During Chunk-based Pre-filling, we process the visual token sequence by dividing it into equal-length chunks. We then compute full attention within each chunk while applying sparse attention between chunks. This approach drastically reduces the overall computational and memory overhead during the pre-filling of vast token sequences. Subsequently, for the decoding stage, our Bi-level KVs Decoding strategy continues to manage the KV cache in chunks. The original KVs generated during pre-filling are designated as dense KVs. We then apply a downsampling operation on the dense KVs of each chunk to obtain sparse KVs. This process ensures that every chunk of the raw video input corresponds to a bi-level KV representation (both dense and sparse). During decoding, we selectively reload either dense or sparse KVs for each video chunk based on its relevance to the specific text query. This approach further optimizes memory usage and enhances the capture of fine-grained information. Based on its innovative design, Video-XL-2 delivers significant advancements with the following key contributions: 1. State-of-the-Art Performance : Video-XL-2 achieve superior performance on various long video understanding benchmarks, outperforming all existing open-source lightweight (7B or 8B) MLLMs. 2. Minimal Memory Footprint : Our model exhibits exceptionally low memory requirements, enabling it to process up to 10,000 frames on a single 80G A100 GPU. 3. Accelerated Inference Speed : Video-XL-2 boasts high inference throughput, capable of processing thousands of frames significantly faster than other comparable open-source MLLMs. 2 Related work Multimodal Large Language Models. The landscape of artificial intelligence has been reshaped by the emergence of Multimodal Large Language Models (MLLMs), which align powerful pre-trained vision encoders with large language models (LLMs) to perform complex vision-language tasks. Early pioneering works primarily focused on images. These models typically follow one of two dominant architectural paradigms. The first, popularized by models like LLaVA liu2023visual_llava and MiniGPT-4 zhu2023minigpt4 , employs a simple projection module (e.g., a linear layer or a shallow MLP) to map visual features into the LLM’s embedding space. The second, exemplified by Flamingo alayrac2022flamingo and BLIP-2 li2023blip2 , utilizes a more sophisticated cross-attention mechanism, often called a querying transformer (Q-Former), to distill visual information into a fixed number of learnable query tokens. The success of these image-based MLLMs naturally inspired their extension to the video domain videollama ; li2023videochat ; maaz2023videochatgpt ; yang2025vidtext ; li2025vid . Initial works such as Video-LLaMA videollama and Video-ChatGPT maaz2023videochatgpt adapted these architectures by treating a video as an ordered sequence of frames. Visual features are extracted from each frame and concatenated, forming a long sequence of visual tokens that is subsequently processed by the LLM. While demonstrating promising video reasoning capabilities, this direct extension exacerbates a fundamental bottleneck: the quadratic complexity of the self-attention mechanism within

Read the complete text