Small Vision-Language Models are Smart Compressors for Long Video Understanding
Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, Saksham Suri, Qi Qian, Shuming Liu, Lemeng Wu, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Chenchen Zhu · 2026
Read Small Vision-Language Models are Smart Compressors for Long Video Understanding by Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, Saksham Suri, Qi Qian, Shuming Liu, Lemeng Wu, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Chenchen Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08120.txt.
Introduction
Fei and 15 colleagues propose Tempo, a query-aware framework that compresses long videos using small vision-language models. Hour-long videos bottleneck multimodal large language models: dense visual streams saturate token budgets and worsen the lost-in-the-middle phenomenon. Sparse sampling or uniform pooling blindly waste fidelity, dropping decisive moments and wasting bandwidth on other backgrounds. Tempo uses a small vision-language model as a local temporal compressor. It casts token reduction as early cross-modal distillation, generating tight, intent-aligned representations in a single forward pass.
Catalog record
| CREATOR | Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, Saksham Suri, Qi Qian, Shuming Liu, Lemeng Wu, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Chenchen Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62915 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08120
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Small Vision-Language Models are Smart Compressors for Long Video Understanding Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, Saksham Suri, Qi Qian, Shuming Liu, Lemeng Wu, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Chenchen Zhu Abstract Adapting Multimodal Large Language Models (MLLMs) for hour-long video understanding is severely bottlenecked by context window limits. Dense visual streams quickly saturate input token budgets and exacerbate the lost-in-the-middle phenomenon. Existing efficiency heuristics, such as sparse sampling or query-agnostic uniform pooling, blindly sacrifice fidelity. They frequently discard transient decisive moments, blur fine-grained evidence, and waste representational bandwidth on irrelevant backgrounds. In this paper, we propose Tempo , an efficient, query-aware framework that compresses long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) to act as a local temporal compressor. It casts visual token reduction as an early cross-modal distillation process, generating compact, intent-aligned video representations in a single forward pass. To enforce strict inference budgets without breaking causality, we introduce Adaptive Token Allocation (ATA) . Exploiting the SVLM’s inherent zero-shot relevance prior and empirical semantic front-loading, ATA acts as a training-free, O ( 1 ) O(1) dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments demonstrate that our compact 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5–16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual token budget, outperforming proprietary baselines such as GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames pushes performance to 53.7 . Crucially, empirical profiling reveals that Tempo frequently compresses hour-long videos to token counts substantially below theoretical computational limits, proving that true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows. † † affiliation: Meta AI † † affiliation: King Abdullah University of Science and Technology (KAUST) † † contribution: Work done at Meta † † contribution: Project lead † † date: August 10, 2026 † † correspondence: [email protected] , [email protected] † † Code: https://github.com/FeiElysia/Tempo † † Project \& Demo: https://FeiElysia.github.io/tempo-page/ Figure 1: Tempo achieves SOTA long video understanding via query-aware Adaptive Token Allocation (ATA). (a) Motivation: Query-agnostic methods either miss transient moments (sparse sampling) or blur details (uniform pooling). Tempo instead utilizes a small vision-language model as a smart compressor for query-aware cross-modal distillation. (b) Mechanism: ATA dynamically allocates high bandwidth (16 tokens/frame) to relevant segment for fine-grained details, while compressing redundant contexts into minimal temporal anchors ( \sim 0.5 tokens/frame) to maintain causality. (c) Result: Leading performance on LVBench. Tempo-6B achieves superior accuracy at extreme compression rates ( i.e. , 4 or 6 tokens/frame), outperforming open-source models and proprietary baselines with a fraction of the context budget. 1 Introduction The advancement of Multimodal Large Language Models (MLLMs) has significantly transformed visual understanding, empowering systems to perform complex semantic analysis over images and short video clips Liu et al. 2023 ; Liu et al. 2024a ; Zhu et al. 2023 ; Li et al. 2024a ; Bai et al. 2025 ; Li et al. 2025 ; Zhang et al. 2024b ; Zhang et al. 2023 . However, scaling these capabilities to hour-long videos remains challenging. The core difficulty lies in the structural mismatch between the massive, continuous visual stream of long videos and the rigidly bounded context windows of downstream LLMs. As temporal duration expands, raw visual tokens quickly overwhelm the input capacity, severely diluting attention mechanisms and causing models to fail at retrieving sparse evidence buried within extensive contexts Liu et al. 2024c . To fit long video understanding into limited contexts, existing methods typically force one of two compromises. A common approach is sparse frame sampling Xu et al. 2024a ; Li et al. 2025 ; Lin et al. 2024a , which reduces compute but inevitably risks skipping the transient yet decisive moments required to answer a specific query. Alternatively, methods retain more frames but apply query-agnostic compression, such as uniform spatiotemporal pooling Maaz et al. 2024 ; Jiang et al. 2025 or token merging Bolya et al. 2022 ; Li et al. 2024b ; Jin et al. 2024 . By compressing without knowing what the user will ask , these heuristics often blur fine-grained evidence in query-critical segments while wasting representational bandwidth on irrelevant backgrounds. In essence, most existing pipelines reduce visual evidence before interacting with the language model, preventing the dynamic allocation of bandwidth to query-critical segments. Even pioneering query-aware approaches ( e.g. , LongVU Shen et al. 2024 ) rely on disjoint auxiliary feature-matching modules, thereby decoupling the routing mechanism from the end-to-end multimodal pipeline. We introduce Tempo , an efficient query-aware framework for long video understanding that natively learns to compress videos for downstream text generation tasks. As its name suggests, Tempo acts as an intelligent temporal compressor that dynamically distributes the rhythm of the video: it allocates high token bandwidth to semantic beats relevant to the query while swiftly fast-forwarding through redundant contexts. Rather than treating visual compression as a purely visual, query-agnostic operation Jiang et al. 2025 ; Li et al. 2024b , Tempo casts this reduction as an early cross-modal semantic distillation process. Concretely, Tempo leverages a Small Vision-Language Model (SVLM) as a local compressor, seamlessly bridging it with an LLM for global understanding and response generation. By prepending the user query to the SVLM input, Tempo performs a preliminary cross-modal distillation pass that produces compact video memory tokens aligned with the user’s intent and is trained end-to-end with standard auto-regressive objectives. A practical challenge is enforcing a strict token budget at inference time ( e.g. , representing a 1024-frame video under an 8K visual token budget) without sacrificing either fine-grained evidence or global causal structure. To this end, we propose Adaptive Token Allocation (ATA) , a training-free inference strategy guided by two key empirical properties of the Tempo architecture. (i) Zero-shot relevance prior and temporal anchors. Inheriting from the base model’s extensive multimodal pre-training, the local compressor exhibits a zero-shot ability to estimate query-video relevance without auxiliary supervision. ATA exploits this prior to allocate budgets segment-wise, enabling an aggressive dynamic compression range (0.5–16 tokens per frame). Crucially, instead of hard pruning, which breaks causality, ATA preserves dense representational bandwidth for relevant segments while compressing redundant contexts into minimal temporal anchors ( i.e. , 4 tokens) to maintain the global storyline. (ii) Semantic front-loading driven by causal attention. Our ablations empirically reveal that under the SVLM’s causal attention, salient visual semantics natively concentrate into the earliest video memory tokens. Consequently, a simple head truncation effectively isolates high-value evidence, avoiding lossy spatial blurring with zero overhead. In summary, our contributions are: • Tempo: an end-to-end, query-aware compression framework for long video understanding. It directly addresses the context window bottleneck by unifying an SVLM-based local compressor and an LLM-based global decoder, performing query-conditioned cross-modal distillation in a single forward pass. • ATA: a training-free, budget-aware inference strategy leveraging the local compressor’s inherent zero-shot relevance prior and semantic front-loading. ATA dynamically dictates the optimal token allocation, preserving fine-grained details for query-critical moments while compressing redundancies into minimal temporal anchors to maintain global causal structure. • Scaling Behaviors: an empirical analysis revealing that optimal resource allocation