FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
Yunzhu Zhang Yu Lu Tianyi Wang Fengyun Rao, Yi Yang Linchao Zhu · 2025
Read FlexSelect: Flexible Token Selection for Efficient Long Video Understanding by Yunzhu Zhang Yu Lu Tianyi Wang Fengyun Rao, Yi Yang Linchao Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00993.txt.
Introduction
Yunzhu Zhang and co-authors teach video models to skip the boring frames in this 2025 paper. Long videos flood video-language models with tokens, most of them redundant, driving cost and memory sky-high. Their method, FlexSelect, ranks video pieces by how strongly a reference layer links them to the question.
A small trained selector then mimics those rankings, discarding filler while keeping meaning. The module plugs into several popular video models without redesign. On four long-video benchmarks it improved accuracy while speeding one model up to ninefold. The authors pitch flexible token choice as the practical route to hour-scale video understanding on real hardware budgets.
Catalog record
| CREATOR | Yunzhu Zhang Yu Lu Tianyi Wang Fengyun Rao, Yi Yang Linchao Zhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00993
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding Yunzhu Zhang Yu Lu Tianyi Wang Fengyun Rao, Yi Yang Linchao Zhu Abstract Long-form video understanding poses a significant challenge for video large language models (VideoLLMs) due to prohibitively high computational and memory demands. In this paper, We propose FlexSelect , a flexible and efficient token selection strategy for processing long videos. FlexSelect identifies and retains the most semantically relevant content by leveraging cross-modal attention patterns from a reference transformer layer. It comprises two key components: (1) a training-free token ranking pipeline that leverages faithful cross-modal attention weights to estimate each video token’s importance, and (2) a rank-supervised lightweight selector that is trained to replicate these rankings and filter redundant tokens. This generic approach can be seamlessly integrated into various VideoLLM architectures, such as LLaVA-Video, InternVL and Qwen-VL, serving as a plug-and-play module to extend their temporal context length. Empirically, FlexSelect delivers strong gains across multiple long-video benchmarks – including VideoMME, MLVU, LongVB, and LVBench. Morever, it achieves significant speed-ups ( e.g., up to 9 × \times on a LLaVA-Video-7B model), highlighting FlexSelect’s promise for efficient long-form video understanding. Project page: https://yunzhuzhang0918.github.io/flex_select . 1 Introduction Long-form video understanding is crucial for applications such as analyzing movies, building multimodal web agents [ 26 ] , assisting in video surveillance tasks. Recent Video Large Language Models (VideoLLMs) [ 1 , 47 , 6 , 21 , 49 , 41 , 19 ] have shown impressive results on short video clips, combining vision and language processing to answer questions or follow instructions about video content. However, extending these models to process long videos presents significant challenges. Long videos yield a substantial volume of visual tokens, often surpassing the context length of transformer-based LLMs. Additionally, processing entire long videos with VideoLLMs leads to excessive computational and memory overhead, making effective long video analysis infeasible. To mitigate this, some recent works [ 8 , 27 ] employ training-based compression modules [ 16 , 45 ] to summarize visual tokens into a compact representation via finetuning on long-video datasets. However, these methods introduce substantial overhead due to additional training. Alternatively, other approaches [ 12 , 29 ] leverage cross-modal attention scores from pre-trained VideoLLMs to rank and prune video tokens without training. While more efficient, these attention-based heuristics often suffer from performance degradation due to inconsistent relevance patterns across transformer layers, which may not reliably reflect the semantic importance of visual tokens for the task. In this paper, we present FlexSelect , a general-purpose framework for efficient long-form video understanding. FlexSelect enables VideoLLMs to focus on the semanticlly relevant visual tokens for the query by identifying and filtering out less informative visual tokens before heavy multimodal reasoning occurs. Crucially, FlexSelect is architecture-agnostic and does not require any modifications or training of the base VideoLLM. It acts as a preprocessor that significantly extends the model’s effective temporal context window without compromising its reasoning ability. A central observation underlying FlexSelect is that well-trained VideoLLMs inherently encode meaningful cross-modal relevance signals within their internal attention maps. In particular, the cross-attention weights between textual queries and visual tokens progressively reflect semantic alignment across transformer layers and typically peak at an intermediate depth (Fig. 1 (a)). Motivated by this, FlexSelect extracts attention scores from an empirically identified optimal reference layer to derive faithful, training-free token importance rankings . These scores enable the selection of semantically critical visual tokens while discarding redundant or irrelevant ones, thus substantially reducing token volume without compromising the model’s reasoning capability. Unlike existing methods that rely on extensive training or oversimplified heuristics, FlexSelect dynamically balances efficiency and performance by exploiting the latent structure of attention patterns within VideoLLMs. Figure 1: (a) Visualization of cross-modal attention maps of LLaVA-Video-7B across layers (user query : "what’s the color of the cup?" ). Attention scores progressively highlight the query-related regions (the cup) with layer depth, and this highlighting is most pronounced at the specific reference layer (layer 19 in example). FlexSelect employs attention scores from this layer to select semantically related visual tokens. (b) VideoMME accuracy and response time (time to generate the first token) of LLaVA-Video-7B. The original model with 64 input frames achieves limited accuracy 64.4% due to inadequate coverage for long video content, while increasing frames will overload the model’s context window, reducing accuracy to 58.5% and slowing response time to 38.2s. FlexSelect improves this by filtering irrelevant tokens, achieving 68.9% accuracy at 512 frames with 9× faster response (4.2s). To reduce the computational cost of processing long videos with VideoLLM layers for token ranking, we introduce a lightweight selector network trained to mimic the reference layer’s ranking. The selector is supervised with a ranking loss and learns to assign higher scores to tokens that the reference model would attend to. This allows for flexible token selection without retraining the full model. Once trained, it efficiently selects the top-ranked visual tokens from long videos, significantly reducing computational overhead. These tokens are then passed into original VideoLLM for final reasoning. In doing so, FlexSelect replicates large model’s attention patterns at a fraction of the computational cost. FlexSelect is a generic approach can be seamlessly integrated into various VideoLLM architectures, such as LLaVA-Video, InternVL and Qwen-VL . FlexSelect serves as a plug-and-play inference module requiring no changes to the base VideoLLM. We evaluate it on four challenging long-video understanding benchmarks—VideoMME, MLVU, LongVB, and LVBench—using VideoLLMs of varying sizes, from 7B to 72B parameters. Experimental results show that FlexSelect achieving significant speed-ups (e.g., up to 9\times faster inference with LLaVA-Video-7B) while maintaining or improving performance. Notably, by filtering out irrelevant content, FlexSelect not only accelerates inference but enhances final answer quality. Contributions. To summarize, our contributions are threefold: • We propose FlexSelect , a flexible and architecture-agnostic framework that extracts faithful cross-modal token relevance from an optimal reference layer in VideoLLMs to select semantically important visual tokens for long-form video understanding. • We design a lightweight rank-supervised selector that mimics the cross-modal attention ranking from a reference model layer, enabling fast and accurate token filtering without modifying or retraining the base VideoLLM. • We demonstrate that FlexSelect is a generic framework applicable to various VideoLLMs, achieving up to \mathbf{9\times} speed-up and improved accuracy on four long-video benchmarks across multiple model scales. 2 Related Works Long-form Video Understanding Current VideoLLMs showcase remarkable video-language understanding abilities. However, it is still challenge to process long-form videos due to the extensive visual tokens. Recent approaches mainly address this by two ways: (1) applying length extrapolation methods (e.g. YARN [ 25 ] ) to VideoLLMs and training on longer sequences [ 5 , 43 ] to support long context input. (2) adopting trainable compression modules to VideoLLM to compress visual content into fewer tokens [ 10 , 18 , 27 , 8 , 13 ] via post-finetuning. Different from these approaches, we introduce a token selector to directly select semantically relevant visual tokens before LLM generation, without training the large-scale VideoLLM, which is more efficient. Attention-based Token Pruning Recent works explore visual token pruning for efficient image-text understanding using cross-modal attention scores. FastV [ 4 ] first identifies visual token inefficiency in LLM processing, pruning tokens via second-layer attention scores but suffering significant accuracy drops. PyramidDrop