R E T A K E : Reducing Temporal and Knowledge Redundancy for Long Video Understanding

Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, Liqiang Nie · 2024

Read R E T A K E : Reducing Temporal and Knowledge Redundancy for Long Video Understanding by Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, Liqiang Nie (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20504.txt.

Description

Abstract Video Large Language Models (VideoLLMs) have made significant strides in video understanding but struggle with long videos due to the limitations of their backbone LLMs. Existing solutions rely on length extrapolation, which is memory-constrained, or visual token compression, which primarily leverages low-level temporal redundancy while overlooking the more effective high-level knowledge redundancy.

Catalog record

CREATORXiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, Liqiang Nie
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

R E T A K E : Reducing Temporal and Knowledge Redundancy for Long Video Understanding Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, Liqiang Nie Abstract Video Large Language Models (VideoLLMs) have made significant strides in video understanding but struggle with long videos due to the limitations of their backbone LLMs. Existing solutions rely on length extrapolation, which is memory-constrained, or visual token compression, which primarily leverages low-level temporal redundancy while overlooking the more effective high-level knowledge redundancy. To address this, we propose R E T A K E , a training-free method with two novel modules DPSelect and PivotKV , to jointly reduce both temporal visual redundancy and knowledge redundancy for video compression. To align with the way of human temporal perception, DPSelect identifies keyframes based on inter-frame distance peaks. To leverage LLMs’ learned prior knowledge, PivotKV marks the keyframes as pivots and compresses non-pivot frames by pruning low-attention tokens in their KV cache. R E T A K E can be plug-and-play adapted to the existing VideoLLMs and enables them to process 8× longer frames (up to 2048), outperforming similar-sized models by 3–5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench. Moreover, by overlapping compression operations with prefilling, R E T A K E introduces only 10% prefilling latency overhead while reducing decoding latency by 20%. Our code is available at https://github.com/SCZwangxiao/video-ReTaKe . 1 Introduction In pursuit of general intelligence, Video Large Language Models (VideoLLMs) [ 16 , 20 , 45 , 24 ] have revolutionized video understanding. However, as a nature extension of Multimodal Large Language Models (MLLMs), they require hundreds of tokens to represent a single image [ 34 , 14 ] , making it impractical to process videos longer than several minutes. Specifically, the context length of common MLLMs [ 34 , 14 ] limits their ability to handle only short videos of around 4 minutes at 240 frames. Figure 1 : R E T A K E effectively compresses video sequence in VideoLLMs, allowing longer perception and improved performance within a fixed memory budget (measured by context length). To empower VideoLLMs for longer videos, Lin et al. [ 20 ] and Li et al. [ 17 ] simply adopt sparse sampling. This approach overlooks the intra-frame spatial details and the inter-frame temporal structure [ 37 ] . To enable dense frame sampling, some methods [ 46 , 29 ] extend the context length of MLLMs through long context post-training. Although they increase performance under long frames, the maximal number of frames is still limited by GPU memory. Therefore, recent methods [ 30 , 13 , 10 , 37 ] compress visual tokens to fit more frames into VideoLLM within existing GPU memory limits, including token sparsification both between and within frames [ 13 ] or using visual token compressors like Q-Former [ 16 ] . However, these methods mainly focus on spatial and temporal visual redundancies [ 36 , 52 ] , which are low-level with high information loss. A promising solution to these limitations is R E ducing T empor A l and K nowledge r E dundancy ( R E T A K E ) for compression. Contrary to low-level temporal redundancy, knowledge redundancy captures high-level abstract patterns or structures that can be inferred from human prior knowledge [ 8 ] , allowing for better compression ratios with smaller loss. The biggest knowledge base in VideoLLMs is the LLM. The attention patterns across LLM layers inherently capture token redundancy, allowing lower-scoring tokens to be dropped without significantly impacting performance [ 49 ] . In VideoLLMs, the biggest knowledge base is the LLM itself. The attention patterns across its layers naturally capture token redundancy, allowing lower-scoring tokens to be dropped without significant performance impact [ 49 ] . However, compression based on knowledge redundancy introduces more computational overhead. To mitigate this, R E T A K E integrates both knowledge and low-level temporal redundancy, allowing VideoLLMs to process significantly more frames within fixed GPU memory constraints and litter computational overhead, thereby enhancing long-video understanding, as illustrated in Figure 1 * * * 2FPS-256 means uniform frame sampling at 2FPS, with a limit 256 frames. For longer videos, FPS is reduced to maintain this limit. . R E T A K E framework comprises two training-free components: DPSelect and PivotKV . To reduce low-level temporal redundancy, DPSelect extracts keyframes before feeding into LLM. Based on methods [ 10 , 7 ] that select keyframes using top-N inter-frame distance, DPSelect identifies additional frames with peak distance as pivot frames, hence the name “Dist Peak Select”. This strategy is more closely aligned with human temporal perception which tracks the peak stimulus to detect motion [ 23 ] . Building on DPSelect, to reduce high-level knowledge redundancy, we develop PivotKV to compress KV cache of video sequences in VideoLLMs. Unlike existing LLM compression techniques that treat all tokens equally [ 49 , 18 ] , PivotKV preserves pivot frames selected by DPSelect, ensuring critical low-level details remain intact. It then compresses non-pivot visual tokens based on their attention scores, which implicitly capture token redundancy identified by the VideoLLM’s high-level multimodal knowledge. We conduct extensive experiments on various long-video understanding benchmarks, including VideoMME [ 5 ] , MLVU [ 51 ] , LongVideoBench [ 38 ] , and LVBench [ 35 ] . Results show that R E T A K E enables VideoLLMs to process 8× more frames (up to 2048), allowing them to achieve a 3%–5% performance gain over models of similar size.. It even surpasses much larger models, such as VideoLLaMA2-72B [ 12 ] , LLaVA-OneVision-72B [ 3 ] , and Oryx-1.5-34B [ 22 ] , on VideoMME, MLVU, and LVBench, respectively. Additionally, R E T A K E introduces only 10% prefilling latency overhead while reducing decoding latency by 20%. Ablation study further validates the complementarity of DPSelect and PivotKV, and R E T A K E ’s superiority against other video compression methods.1 In summary, our contributions are threefold: • To our best knowledge, the training-free R E T A K E is the first to jointly model temporal and knowledge redundancy for long video compression, enabling VideoLLMs to process 8× longer frames (up 2048). • We propose a novel keyframe selection method DPSelect to reduce low-level temporal redundancy, and a novel token compression method PivotKV to reduce high-level knowledge redundancy in long videos. • R E T A K E introduces only 10% prefilling latency overhead while reducing decoding latency by 20%. It helps existing VideoLLMs outperform similar-sized VideoLLMs by 3%-5%, setting new state-of-the-art benchmarks. 2 Related Work 2.1 Video Large Language Models Most VideoLLMs build on image MLLMs and are further trained on video-text pairs to capture temporal relationships. A VideoLLM typically comprises a vision foundational model (VFM) [ 26 ] , an LLM [ 4 ] , and a connector linking them. The VFM extracts visual features, while the connector maps them into a format the LLM can process for video understanding. VideoLLMs fall into two main types based on the connector: concatenation-based and Q-Former-based. Concatenation-based models [ 24 , 20 , 21 ] are simple and effective but suffer from exponentially increasing

Read the complete text