Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding
Wenhui Tan \sthanks Work performed when Wenhui Tan was a research intern at Xiaomi Inc, Ruihua SongGaoling School of Artificial IntelligenceRenmin University of ChinaBeijing, China, Jiaze Li, Jianzhong Ju, Zhenbo LuoMiLM PlusXiaomi Inc.Beijing, China · 2026
Read Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding by Wenhui Tan \sthanks Work performed when Wenhui Tan was a research intern at Xiaomi Inc, Ruihua SongGaoling School of Artificial IntelligenceRenmin University of ChinaBeijing, China, Jiaze Li, Jianzhong Ju, Zhenbo LuoMiLM PlusXiaomi Inc.Beijing, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11359.txt.
Introduction
Wenhui Tan, Ruihua Song, Jiaze Li, Jianzhong Ju, and Zhenbo Luo pick frames for video AI. Their 2026 preprint is named Think-Clip-Sample, a slow-fast method. Multimodal large language models have advanced video understanding fast. Long videos still strain them, for two clear reasons.
Compute cost caps how many frames a model can read. Poor frame choice wastes that small budget on bland shots. The team's method picks frames in a slow-fast rhythm. Slow thought plans where the key content may sit. Fast sampling then grabs frames that match the plan. Better frames mean better answers on long videos. The work came from Renmin University and Xiaomi teams.
Catalog record
| CREATOR | Wenhui Tan \sthanks Work performed when Wenhui Tan was a research intern at Xiaomi Inc, Ruihua SongGaoling School of Artificial IntelligenceRenmin University of ChinaBeijing, China, Jiaze Li, Jianzhong Ju, Zhenbo LuoMiLM PlusXiaomi Inc.Beijing, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding Wenhui Tan \sthanks Work performed when Wenhui Tan was a research intern at Xiaomi Inc, Ruihua SongGaoling School of Artificial IntelligenceRenmin University of ChinaBeijing, China, Jiaze Li, Jianzhong Ju, Zhenbo LuoMiLM PlusXiaomi Inc.Beijing, China Abstract Recent progress in multi-modal large language models (MLLMs) has significantly advanced video understanding. However, their performance on long-form videos remains limited by computational constraints and suboptimal frame selection. We present Think-Clip-Sample (TCS), a training-free framework that enhances long video understanding through two key components: (i) Multi-Query Reasoning , which generates multiple queries to capture complementary aspects of the question and video; and (ii) Clip-level Slow-Fast Sampling , which adaptively balances dense local details and sparse global context. Extensive experiments on MLVU, LongVideoBench, and VideoMME demonstrate that TCS consistently improves performance across different MLLMs, boosting up to 6.9% accuracy, and is capable of achieving comparable accuracy with 50% fewer inference time cost, highlighting both efficiency and efficacy of TCS on long video understanding. † † footnotetext: 🖂 Corresponding authors: Ruihua Song and Zhenbo Luo. 1 Introduction Understanding long-form videos has become a central challenge in advancing multi-modal large language models (MLLMs) [ 11 , 17 , 8 ] . A long video often consists of thousands of frames, even with a low frame-per-second (FPS) sampling rate, leading to prohibitive computational costs. Mainstream MLLMs typically adopt uniform frame sampling [ 19 , 1 , 3 ] , but this strategy treats all frames equally, regardless of their informativeness, thus resulting in subpar performance. To mitigate this issue, several methods have been proposed to select more informative frames, e.g., Q-Frame [ 24 ] introduces adaptive frame selection and multi-resolution scaling tailored to the video content and the specific query. It proposes a CLIP-based [ 12 ] similarity with the Gumbel-Max trick for efficient selection, allowing Video-LLMs to process more relevant frames without exceeding computational limits. AKS [ 16 ] formulates selection as an optimization problem balancing query relevance and video coverage, and provides an adaptive algorithm to approximate the optimal solution. Despite these advances, existing approaches face two fundamental challenges. First, they rely solely on direct question-frame similarity, assuming the question adequately represents all information needs. However, questions are often abstract and incomplete. For instance, a question like “ Which team won in the end? (A) Team in black clothes; (B) Team in white clothes ” only mentions subjects while omitting crucial details about actions and context. Directly feeding such questions to CLIP often retrieves frames showing players without capturing game-decisive moments. Second, similarity-based sampling tends to produce unbalanced frame distributions. Algorithms may duplicate spikes of high similarity frames, while neglecting informative regions with moderate similarity scores and global context. This results in sparse coverage that misses important information for comprehensive video understanding. In this paper, we propose Think-Clip-Sample (TCS) , a training-free method designed to address these challenges. (1) To enhance diversity and ensure broader coverage, TCS first thinks of multiple queries [ 6 , 14 , 5 , 15 ] . Instead of relying on a single question, we prompt the MLLM to generate multiple queries from different perspectives (e.g., objects, scenes, actions). These multi-view queries encourage the MLLM to capture complementary information for answering the question across diverse video segments. (2) To avoid sparse and uneven sampling, we propose a Clip-level Slow-Fast Sampling strategy: Given a total frame budget K , TCS first identify high-similarity clips rather than isolated frames. Then, a larger portion of K is allocated to these compact yet informative clips, denoted as slow sampling , while the remaining frames are uniformly sampled from non-clipped regions (fast sampling) to provide global coverage. This balanced allocation ensures both fine-grained detail and a global context. We evaluate TCS on two base MLLMs, Qwen2-VL-7B [ 19 ] and MiMo-VL-7B [ 21 ] , across three challenging benchmarks: LongVideo-Bench [ 20 ] , MLVU [ 25 ] , and VideoMME [ 4 ] . Experimental results demonstrate that TCS consistently improves performance of the base MLLMs under the same sampling budget, achieving up to 6.9% accuracy improvement on MLVU, while reducing computation time by over 50% on Qwen2-VL-7B with comparable performance, highliting both efficiency and efficacy of TCS for long video understanding. Figure 1: Overview of our proposed method Think-Clip-Sample (TCS): (i) Given a question on a long video, TCS first thinks of queries from different perspectives, which are used with CLIP to retrieve frames with both high relevance and broad coverage. (ii) Instead of sampling frames with highest similarity scores, TCS identifies high-relevance clips , and then (iii) applies Slow-Fast Sampling to allocate more frames on informative clips (yellow) and distribute the remainder across non-clip regions (gray), preserving both local detail and global context. Our main contributions are threefold: • We propose Multi-Query Reasoning , to automatically generate multiple queries from the question and video, enabling the retrieval of diverse and complementary frames from different perspectives such as objects, actions, and scenes. • We introduce Clip-level slow-fast sampling strategy, which first identifies high-similarity clips and allocates a larger portion of the frame budget to these regions, and uniformly sampling the remainder from non-clipped regions, to ensure both fine-grained detail and global coverage. • Experimental results demonstrate that TCS significantly improves long video understanding across multiple MLLMs and benchmarks, achieving up to 6.9% accuracy gain, and reducing inference time by over 50% with comparable performance, demonstrating both efficiency and efficacy. 2 Method 2.1 Overview This work focuses on video understanding through video question answering tasks. Given a video V\in\mathbb{R}^{T\times H\times W\times 3} with T frames and (H\times W) resolution, and a multi-option question Q , the goal of the MLLM is to select the correct answer among the options. For long videos (e.g., longer than 10 minutes), T can be very large and infeasible to process directly. Thus, selecting a subset of K frames, where K\ll T , is crucial for efficient video understanding. A straightforward approach is to uniformly sample K frames across the video. Although this ensures temporal coverage, it often results in sparse observations that miss crucial visual evidence. To address this, a common strategy is to compute similarity scores \mathbf{s}=\{s_{i}\}_{i=1}^{T} between the question and frames using a pre-trained vision-language models (VLMs) [ 22 , 10 , 23 , 18 , 9 ] , such as CLIP [ 12 ] , and then sample frames according to \mathbf{s} . However, this pipeline presents two key challenges: (1) how to obtain better similarity scores \mathbf{s} , and (2) how to design an effective sampling strategy given \mathbf{s} . We address these in Section 2.2 and 2.3 , respectively. 2.2 Reasoning for Multi-Perspective Queries Previous work typically computes similarity scores directly between the question and video frames V with VLMs such as CLIP. However, these VLMs are designed to compare visual descriptions with images, rather than natural language questions. Furthermore, a single question may only highlight one aspect of the visual content, resulting in incomplete frame retrieval. For example, given a question such as “ Which team won in the end? (A) Team in Black Clothes (B) Team in White Clothes ”, CLIP would retrieve all the frames where the players are present, lacking core relevance to the