CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks

Yanan Wang, Julio Vizcarra, Zhi Li, Hao Niu, Mori Kurokawa · 2025

Read CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks by Yanan Wang, Julio Vizcarra, Zhi Li, Hao Niu, Mori Kurokawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13609.txt.

Introduction

Five researchers break video questions into steps to teach models finer reasoning in a 2025 paper. Video language models learn from high level text pairs and often miss object level detail. Step by step thought, tied to the objects on screen, is the missing skill the team targets.

Their frame, CoTasks, splits hard questions from two known data sets into four base tasks. The steps cover finding frames, tracking entities, and reading space and time relations. Two video models gained clear score lifts on a standard test after the training. Causal, time based, and descriptive answers gained the most, which supports the step by step design.

Catalog record

CREATORYanan Wang, Julio Vizcarra, Zhi Li, Hao Niu, Mori Kurokawa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks Yanan Wang, Julio Vizcarra, Zhi Li, Hao Niu, Mori Kurokawa Abstract Despite recent progress in video large language models ( VideoLLMs ), a key open challenge remains: how to equip models with chain-of-thought ( CoT ) reasoning abilities grounded in fine-grained object-level video understanding. Existing instruction-tuned models, such as the Qwen and LLaVA series, are trained on high-level video-text pairs, often lacking structured annotations necessary for compositional, step-by-step reasoning. We propose CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks , a new framework that decomposes complex video questions of existing datasets ( e.g. , NeXT-QA, STAR) into four entity-level foundational tasks: frame localization, entity tracking, spatial and temporal relation extraction. By embedding these intermediate CoT-style reasoning steps into the input, CoTasks enables models to explicitly perform object-centric spatiotemporal reasoning. Experiments on the NeXT-QA benchmark show that CoTasks significantly enhance inference performance: LLaVA-video-7B improves by +3.3 points in average GPT-4 evaluation score, and Qwen2.5-VL-3B gains +17.4 , with large boosts in causal ( +14.6 ), temporal ( +10.9 ) and descriptive ( +48.1 ) subcategories. These results demonstrate the effectiveness of CoTasks as a structured CoT-style supervision framework for improving compositional video reasoning. 1 Introduction Figure 1: Comparison of LLaVA-video-7B with and without CoTasks. Given a complex video question, direct inference yields a shallow answer (“Sit”), whereas CoTasks guides the model through entity-aware reasoning, resulting in a more grounded and descriptive response. Video Large Language Models (VideoLLMs) are rapidly gaining importance in applications such as interactive video QA systems , including ChatGPT OpenAI 2023 and Gemini DeepMind 2023 , and embodied agents , including OK-Robot Liu et al. 2024c and DynaMem Liu et al. 2024b . These models aim to understand complex, dynamic visual scenes and generate coherent answers or decisions from high-level natural language queries. Despite recent advances, a key open challenge remains: how to equip these models with structured reasoning abilities that reflect the step-by-step understanding humans apply when interpreting real-world videos. Current state-of-the-art VideoLLMs, including the Qwen ( Bai et al. 2023 ; Wang et al. 2024 ; Bai et al. 2025 ) and LLaVA ( Liu et al. 2023 ; Zhang et al. 2024 ) families, are predominantly trained using high-level video instruction tuning. These models perform reasonably well on surface-level tasks, but often fail in scenarios that require nuanced object-level and temporal reasoning. As shown in Figure 1 , given a complex video-based question, models like LLaVA-video-7B tend to produce shallow answers such as “Sit” failing to ground the response in the visual context or reflect the full narrative. We hypothesize that this limitation arises from the absence of intermediate supervision signals , which guide the model to reason about where , who/what , and how entities interact over time. Inspired by Chain-of-Thought (CoT) prompting ( Wei et al. 2022 ) , we argue that injecting structured, step-wise visual cues can bridge this gap. To this end, we introduce CoTasks , a new framework for Chain-of-Thought based Video Instruction Tuning Tasks . CoTasks decomposes high-level questions from existing video QA datasets ( e.g. , NeXT-QA ( Xiao et al. 2021 ) , STAR ( Wu et al. 2021 ) ) into four foundational reasoning subtasks: (1) frame localization , (2) entity tracking , (3) spatial relation extraction , and (4) temporal relation extraction . By embedding these intermediate CoT-style reasoning steps into the input, CoTasks enables models to explicitly perform object-centric spatiotemporal reasoning. As a result, giving a high-level question, the model can ground and compose visual evidence step-by-step before producing an answer. As visualized in Figure 1 , our approach transforms the vague output of standard VideoLLMs into a much richer response that closely mirrors human understanding. We evaluate CoTasks using three VideoLLMs, Qwen2.5-VL-3B , Qwen2.5-VL-7B , and LLaVA-video-7B on the NeXT-QA and STAR benchmarks. To assess the upper bound of performance gains, we prompt models with the original question augmented by the ground-truth answers of the CoTasks subtasks. This allows us to isolate how much structured CoT-style context can contribute to final answer quality. Our inference-time augmentation strategy requires no architectural changes or re-training. CoTasks significantly boosts performance across all models. Notably, Qwen2.5-VL-3B—a lightweight model—achieves a +17.4 point gain in average GPT-4 evaluation score, with large improvements in causal ( +14.6 ), temporal ( +10.9 ), and descriptive ( +48.1 ) reasoning categories. These results demonstrate the effectiveness of CoTasks as a lightweight yet powerful mechanism for enhancing structured video reasoning. Moreover, our findings highlight the promise of using CoTasks as an instruction-tuning curriculum for building high-performance, resource-efficient VideoLLMs—a key requirement for on-device, real-world applications. 2 Problem setup In this work, we focus on constructing a video instruction tuning dataset grounded in chain-of-thought (CoT) reasoning. Given object-centric video question answering (VideoQA) tasks such as NeXT-QA and STAR—which provide fine-grained annotations including object bounding boxes, intra-frame spatial relations, and inter-frame temporal relations—we reformulate the original multiple-choice questions into open-ended, free-form answering tasks. Furthermore, we augment each complex visual reasoning question with four foundational CoTasks: frame localization, object tracking, spatial relation extraction, and temporal relation extraction. The primary objective is to investigate whether these CoTasks can enhance the reasoning capabilities of state-of-the-art videoLLMs. 3 Related work Multimodal Large Language Models (MLLMs). MLLMs, such as LLaVA Liu et al. 2023 , Qwen2-VL Bai et al. 2023 , and LLaMA3-Vision meta 2024 , have extended language models into visual domains by integrating (1) a vision encoder Radford et al. 2021 ; Tschannen et al. 2023 to extract features from images or video frames, (2) a projection module to align visual features with the language embedding space, and (3) a language model backbone for multimodal reasoning. Recent works, including Video-ChatGPT Maaz et al. 2024 , LLaVA-Video Zhang et al. 2024 , Qwen2.5-VL Bai et al. 2025 , and AdaReTaKe Wang et al. 2025 , extend MLLMs to videos by compressing and aligning frame sequences to enable long-context understanding. However, these models primarily focus on high-level instruction tuning without decomposing complex queries into structured sub-tasks—such as frame localization, object tracking, and spatiotemporal relation reasoning—limiting their ability to handle detailed object interactions or causal inference in dynamic scenes. In contrast, our proposed CoTasks framework addresses this limitation by decomposing high-level video QA into four foundational sub-tasks: entity-based frame localization, object tracking, spatial relation extraction, and temporal relation extraction. These sub-tasks serve as intermediate, Chain-of-Thought (CoT)-style reasoning steps that inject structured guidance into video instruction tuning and enhance fine-grained spatiotemporal reasoning. Video Instruction Tuning dataset. Several large-scale video instruction datasets have recently been proposed to improve VideoLLMs’ ability to follow complex visual-language instructions. LLaVA-Video-178K Zhang et al. 2024 is a synthetic dataset of 178K video samples, each annotated with detailed captions and open- and closed-ended questions. Generated using GPT-4 and human feedback, it supports instruction tuning for models like LLaVA-Video but focuses on high-level annotations without explicit intermediate reasoning structures. VideoInstruct-100K Maaz et al. 2024 consists of 100K video-instruction pairs with spatial, temporal, and reasoning-oriented questions, used to train Video-ChatGPT. While high-quality, it lacks structured object-centric decomposition, making it difficult to model fine-grained spatiotemporal reasoning. Video-STaR (VSTaR-1M) Zohar et al. 2025 leverages diverse video sources (e.g., Kinetics-700 Carreira et al. 2019 , STAR Wu et al.

Read the complete text