FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding

Janghoon Cho, Jungsoo Lee, Munawar Hayat, Fatih Porikli, Sungha Choi, [4pt] Qualcomm AI Research · 2025

Read FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding by Janghoon Cho, Jungsoo Lee, Munawar Hayat, Fatih Porikli, Sungha Choi, [4pt] Qualcomm AI Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00141.txt.

Introduction

Janghoon Cho, Jungsoo Lee, and Munawar Hayat present FLoC, a visual token compression method that makes long video understanding by multimodal models affordable. Extended videos generate overwhelming numbers of visual tokens, which limits how much video such models can process. Facility location, a principled selection objective, picks a compact subset of tokens that stays representative and diverse within a fixed budget.

Selected tokens carry the video's varied content without near-duplicates wasting context. Multimodal AI readers get a scalable preprocessing step for hour-scale video. Compression quality decides whether long video models reason about the whole story or only its beginning.

Catalog record

CREATORJanghoon Cho, Jungsoo Lee, Munawar Hayat, Fatih Porikli, Sungha Choi, [4pt] Qualcomm AI Research
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding Janghoon Cho, Jungsoo Lee, Munawar Hayat, Fatih Porikli, Sungha Choi, [4pt] Qualcomm AI Research Abstract Recent studies in long video understanding have harnessed the advanced visual-language reasoning capabilities of Large Multimodal Models (LMMs), driving the evolution of video-LMMs specialized for processing extended video sequences. However, the scalability of these models is severely limited by the overwhelming volume of visual tokens generated from extended video sequences. To address this challenge, this paper proposes FLoC, an efficient visual token compression framework based on the facility location function, a principled approach that swiftly selects a compact yet highly representative and diverse subset of visual tokens within a predefined budget on the number of visual tokens. By integrating the lazy greedy algorithm, our method achieves remarkable efficiency gains by swiftly selecting a compact subset of tokens, drastically reducing the number of visual tokens while guaranteeing near-optimal performance. Notably, our approach is training-free, model-agnostic, and query-agnostic, providing a versatile solution that seamlessly integrates with diverse video-LLMs and existing workflows. Extensive evaluations on large-scale benchmarks, such as Video-MME, MLVU, and LongVideoBench, demonstrate that our framework consistently surpasses recent compression techniques, highlighting not only its effectiveness and robustness in addressing the critical challenges of long video understanding, but also its efficiency in processing speed. 1 Introduction Figure 1: Performance (Average relative accuracy compared to full token usage) versus compression time (log-scale) for a number of compression algorithms. Details are described in Section . With the recent emergence of Large Language Models (LLMs) in natural language processing, there has been a surge of interest in extending their capabilities to the visual domain Achiam et al. 2023 . By utilizing the visual embeddings as token inputs to the LLMs, referred to as visual tokens, these Large Multimodal Models (LMMs) have already demonstrated their performances surpassing human-level accuracy on vision tasks, such as visual question answering Liu et al. 2024 ; Fang et al. 2024 ; Team et al. 2023 . More recently, the research focus has shifted towards enabling these models to understand video sequences Lin et al. 2023 , giving rise to video-LMMs Song et al. 2024 ; Xue et al. 2024 ; Wang et al. 2024a ; Balazevic et al. 2024 . Such models not only excel in tasks like captioning Krishna et al. 2017 ; Xu et al. 2015 ; Vinyals et al. 2015 , event detection Xu et al. 2019 ; Shou et al. 2021 , and action recognition Zhao et al. 2017 ; Simonyan & Zisserman 2014 , but also show significant potential in various real-world applications, including surveillance through CCTV systems, immersive experiences in smart glasses, and autonomous navigation for mobile robots. Despite this progress, long video understanding remains particularly challenging due to the explosive growth in the number of visual tokens as the video sequence length increases Xue et al. 2024 ; Fu et al. 2024 . When dealing with high-resolution or long-duration videos (e.g., 4K content), it becomes computationally infeasible to process every token end-to-end, especially given that most LLM-based architectures support input contexts of only 4K to 32K tokens. This limitation is exacerbated in real-world scenarios: for instance, continuous CCTV footage can span days or weeks, smart glasses may capture extended, first-person video streams, and mobile robots frequently operate in dynamic environments requiring real-time video analysis. Consequently, the gap between human-level performance and current model capabilities still exists, highlighting the complexity and significance of this research direction. To tackle the issue of handling long video sequences, visual token compression is indispensable. In practice, when examining consecutive frames of a video, many tokens share highly redundant information unless there is a substantial scene change Potapov et al. 2014 . Eliminating these redundancies often does not harm the downstream performance, while excessively pruning tokens could lead to the loss of critical information. It is therefore critical to strike a delicate balance in token compression to minimize information loss. Previous approaches to selecting representative visual tokens often relied on filtering out temporally redundant tokens or frames Shen et al. 2024 ; Tao et al. 2024 or clustering techniques to extract representative information from each cluster Wang et al. 2024b ; Shang et al. 2024 ; Zhang et al. 2024a . While these methods may work at a reasonable level, they often fall short in capturing the full diversity needed to interpret complex visual scenes. Consider a scenario where a user wearing smart glasses searches for car keys in a cluttered room. Visual tokens representing the small object of interest (the keys) occur infrequently and sparsely within the video sequence, whereas tokens depicting general scenery, such as furniture or background, appear repeatedly and redundantly. In this setting, clustering-based approaches are likely to fail in capturing rare but important tokens—such as those corresponding to the keys—since they primarily focus on densely populated regions in the feature space. Therefore, a visual token compression algorithm that simultaneously ensures representativeness and diversity is essential to effectively retain these critical but sparse visual cues. Figure 2: Overview of the proposed framework for selecting a visual token subset. Our method compresses the visual tokens extracted by a visual encoder from input video sequences into a diverse and representative subset within a given budget. The selected visual tokens are then concatenated with text tokens and fed into the video-LMM. Since our method is training-free and model-agnostic, it can be seamlessly integrated into any video-LMM in a plug-and-play manner. In order to overcome these limitations, we propose a novel visual token compression algorithm based on the facility location function Lin & Bilmes 2011 ; Lin et al. 2009 . Our approach interprets token selection through the lens of submodular optimization, ensuring that the selected set of tokens covers all original tokens under a given budget constraint. Specifically, each subset considers the similarity between its subset and the entire tokens, enabling to include diverse information of the entire video sequence. While finding the optimal subsets in this manner is known to be a NP-hard problem, we sidestep the computational overhead by utilizing the lazy greedy algorithm Minoux 1978 , enabling to select the visual tokens with minimal computational overheads. As a result, the chosen tokens are both representative and diverse, effectively preserving essential information for video understanding tasks. Our experiments on benchmarks such as Video-MME, and LongVideoBench Zhou et al. 2024 ; Wu et al. 2025 demonstrate the superiority of our method over existing approaches. The remainder of this paper is organized as follows. In Section , we provide a comprehensive review of related work. Section details our proposed facility location-based algorithm for visual token compression. Experimental settings and results are presented in Section , and we conclude in Section by summarizing our key findings and discussing potential future directions. 2 Related Work Sampling / Pooling A common and straightforward strategy to deal with the abundance of visual tokens in long video sequences is to reduce the input size via pooling or sampling Potapov et al. 2014 ; Cai et al. 2024 ; Qu et al. 2024 ; Wu 2024 . For instance, uniform sampling of frames or pooling across spatial/temporal dimensions can substantially cut down the computational overhead and memory usage.

Read the complete text