ThinkVideo: High-Quality Reasoning Video Segmentation with Chain of Thoughts

Shiu-hong Kao, Yu-Wing Tai, Chi-Keung Tang · 2025

Read ThinkVideo: High-Quality Reasoning Video Segmentation with Chain of Thoughts by Shiu-hong Kao, Yu-Wing Tai, Chi-Keung Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18561.txt.

Introduction

Shiu-hong Kao, Yu-Wing Tai, and Chi-Keung Tang present a study from twenty twenty-five on picking out objects in video from veiled text prompts. A user may name a thing in a roundabout way that shifts with time. Past tools fine-tune big vision language models, yet they trip on such time bound prompts. Their frame, CoT-RVS, uses step by step thought at test time.

For each frame, the frame names things that could fit the prompt in sense. It then picks a key frame where each thing shows most clear. No training is needed, so closed models can drive it. The frame can swap its target mid stream when a better one shows. Tests on plain and veiled prompts beat past tools by clear gaps.

Catalog record

CREATORShiu-hong Kao, Yu-Wing Tai, Chi-Keung Tang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18561

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ThinkVideo: High-Quality Reasoning Video Segmentation with Chain of Thoughts Shiu-hong Kao, Yu-Wing Tai, Chi-Keung Tang Abstract Reasoning Video Object Segmentation is a challenging task, which generates a mask sequence from an input video and an implicit, complex text query. Existing works probe into the problem by finetuning Multimodal Large Language Models (MLLM) for segmentation-based output, while still falling short in difficult cases on videos given temporally-sensitive queries, primarily due to the failure to integrate temporal and spatial information. In this paper, we propose ThinkVideo , a novel framework which leverages the zero-shot Chain-of-Thought (CoT) capability of MLLM to address these challenges. Specifically, ThinkVideo utilizes the CoT prompts to extract object selectivities associated with particular keyframes, then bridging the reasoning image segmentation model and SAM2 video processor to output mask sequences. The ThinkVideo framework is training-free and compatible with closed-source MLLMs, which can be applied to Reasoning Video Instance Segmentation. We further extend the framework for online video streams, where the CoT is used to update the object of interest when a better target starts to emerge and becomes visible. We conduct extensive experiments on video object segmentation with explicit and implicit queries. The results show that ThinkVideo significantly outperforms previous works in both cases, qualitatively and quantitatively. Code will be released at github.com/DanielSHKao/ThinkVideo . Figure 1: ThinkVideo is a novel framework for Reasoning Video Segmentation utilizing the zero-shot Chain-of-Thought (CoT) capability of pretrained multimodal Large Language Models, and correctly segments the fast-moving player before, during, and after his three-point shot after CoT, given a temporally sensitive query “ Which player makes a three-point shot in this basketball game ” which requires both spatial and temporal reasoning. Refer to our supplemental videos for showcase demos. 1 Introduction Reasoning Video Object Segmentation (Reasoning VOS) presents a formidable challenge in the field of computer vision, requiring the generation of a mask sequence from an input video alongside an implicit and often complex text query [ 82 , 4 ] . Unlike traditional vision-language tasks that directly associate visual data with textual descriptions, Reasoning VOS requires advanced cognitive capabilities given the implicit nature of the queries. This task is particularly demanding due to the dynamic nature of video data, where factors such as temporally sensitive queries, occlusions, or rapid object movements can complicate the segmentation process. Existing approaches of reasoning segmentation focus on finetuning the multimodal Large Language Models (MLLMs) for segmentation-based tokens. While achieving good results in Reasoning VOS, they require time-consuming finetuning process and still suffer from low accuracy when temporal reasoning is required. An implicit and time sensitive query, “Which player makes a three-point shot in this basketball game?” , for instance, is more challenging than “Which player is wearing black?” ( Figure 1 ). A query is time or temporally sensitive if the relevant video objects with the specified action context ( e.g., three-point shot), do not appear uniformly over time but rather are concentrated at certain intervals, requiring high-level temporal reasoning to correctly find them to the satisfaction of the query requirement. Even though reasoning segmentation has achieved impressive performance on images [ 34 , 31 , 41 ] and videos [ 4 ] , current results are far from perfect in the video domain, showcasing failure in integrating temporal information with spatial and textual context in satisfying complex and time-sensitive queries. To address these challenges, we introduce ThinkVideo , an innovative framework designed to leverage the zero-shot Chain-of-Thought (CoT) capabilities of pre-trained MLLMs ( Figure 1 ). Instead of directly predicting the mask tokens, ThinkVideo breaks down the reasoning process into a sequence of “thinking” steps with multiple synthetic pairs of questions and answers. This thinking procedure will generate instance-level selectivities linked to particular keyframes, then bridging the reasoning image segmentation model with the SAM2 [ 60 ] video processor to generate coherent mask sequences. This approach not only enhances the model’s ability to reason about object relationships and movements but also allows for more accurate segmentation in complex video environments. One of the standout features of the ThinkVideo framework is its training-free nature, making it compatible with open-source and closed-source MLLMs. We leverage the state-of-the-art Gemma3 [ 65 ] and GPT-4o [ 29 ] as the “thinking” agent in our ThinkVideo framework to generate keyframe selectivity. To evaluate the effectiveness of ThinkVideo, we conduct comprehensive experiments on referring VOS datasets ( e.g., MeViS [ 20 ] and Refer-DAVIS-17 [ 58 ] ) and reasoning VOS datasets ( e.g., ReVOS [ 82 ] and ReasonVOS [ 4 ] ). We demonstrate that ThinkVideo significantly outperforms existing referring and reasoning VOS approaches, qualitatively and quantitatively, with its more general formulation based on Reasoning Video Instance Segmentation (Reasoning VIS), which can output multiple instance-level mask sequences that satisfy the user query. In addition, we propose a variant version of ThinkVideo for online video streams, where CoT is used to update the target object when a new object starts to appear in the middle of the video that aligns better with the text query. In conclusion, our contributions include: • We propose ThinkVideo, a zero-shot-CoT framework for Reasoning VIS and VOS. • ThinkVideo significantly outperforms existing SOTA qualitatively and quantitatively. • We extend ThinkVideo to process online video streams. 2 Related Work Video segmentation. Video object segmentation (VOS) tracks objects in a given video and output corresponding mask sequences. Early VOS approaches [ 47 , 10 ] rely on online first-frame finetuning, thereby constrained to specific applications because of the high time cost. To speed up inference, follow-up works adopt efficient online learning algorithm [ 61 , 7 ] , MRF graph inference [ 5 ] , temporal CNN [ 81 , 27 ] , capsule routing [ 24 ] , tracking [ 30 , 68 , 57 , 12 ] , embedding learning [ 83 , 66 , 13 ] and space-time matching [ 28 , 54 , 16 ] . Recent VOS methods allow various control modalities, such as categories [ 14 , 70 , 78 , 89 , 90 , 93 ] , points [ 60 , 59 , 26 ] , segmentation masks [ 15 , 17 , 55 , 57 , 60 ] , or text query [ 8 , 20 , 75 , 62 , 84 , 37 , 76 , 48 , 18 ] , making the segmentation more flexible and interactive. Specifically, building on the advanced development of LLM, Reasoning VOS focuses on generating mask sequences from a highly implicit and complex text query. VISA [ 82 ] first selects a keyframe and follows the core idea of LISA [ 34 ] by generating a segmentation-oriented token with finetuned image-based MLLM. Video-LISA [ 4 ] and HyperSeg [ 71 ] encode the video input in a multi-grained manner and concatenate the text query, feeding both to the LLM for segmentation-based outputs. However, none of the prior works incorporate adequate consideration of CoT reasoning ability for spatial and temporal information in Reasoning VOS. Our ThinkVideo is the first zero-shot-CoT framework compatible with pre-trained MLLM and image segmentation module, making it flexible to a boarder audience. Chain-of-Thought MLLM. Drawing inspiration from the exceptional reasoning capabilities of Large Language Models (LLMs), Multi-modal Large Language Models (MLLMs)

Read the complete text