VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning

Chenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, Jiaqi Wang · 2026

Read VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning by Chenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, Jiaqi Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15724.txt.

Introduction

Chenglin Li and a large team build video models that use tools in a 2026 arXiv paper. Long-form video remains a hard problem for video language models. Most models reason over uniformly sampled frames. That static approach weakens timing: they cannot pinpoint when things happen. It also loses much information in long videos. The team's system is called VideoThinker. It is built as an agentic VideoLLM.

Agentic tools let the model explore a video adaptively. The tools include temporal retrieval, spatial zoom, and temporal zoom. Guided by a language model, the system picks where to look closer. That mirrors how a person scans a long recording. For tasks like surveillance review or lecture search, active looking beats uniform sampling.

Catalog record

CREATORChenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, Jiaqi Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-15724

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning Chenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, Jiaqi Wang Abstract Long-form video understanding remains a fundamental challenge for current Video Large Language Models (VideoLLMs). Most existing models rely on static reasoning over uniformly sampled frames, which weakens temporal localization and leads to substantial information loss in long videos. Agentic tools, such as temporal retrieval, spatial zoom, and temporal zoom, offer a natural way to overcome these limitations by enabling adaptive exploration of key moments. However, constructing agentic video-understanding data requires models that already possess strong long-video comprehension, creating a circular dependency. We address this challenge with VideoThinker, an agentic VideoLLM trained entirely on synthetic tool-interaction trajectories. Our key idea is to convert videos into rich captions and employ a powerful agentic LLM to generate multi-step tool-use sequences in the caption space. These trajectories are subsequently grounded back to video by replacing captions with actual frames, yielding a large-scale interleaved video–tool reasoning dataset without requiring any long-form understanding from the underlying VideoLLM. Training on this synthetic agentic dataset equips VideoThinker with dynamic reasoning capabilities, adaptive temporal exploration, and multi-step tool use. Remarkably, VideoThinker significantly outperforms both caption-only LLM agents and strong VideoLLM baselines across long-video benchmarks, demonstrating the effectiveness of tool-augmented synthetic data and adaptive retrieval–zoom reasoning for long-form video understanding. Figure 1 : Comparison of VideoThinker with VideoLLMs and LLM agents. VideoThinker excels at interleaved video reasoning on long videos, using agentic tools to iteratively perceive and reason over video frames step by step. 1 Introduction Video understanding is a fundamental task that requires reasoning over both spatial and temporal dimensions of videos ( 41 ; 22 ; 3 ; 17 ; 14 ) . Recent advances in Video Large Language Models (VideoLLMs) have achieved remarkable progress in this area ( 16 ; 45 ; 19 ; 13 ; 1 ; 48 ) . Despite these advances, existing models still struggle with long-form videos ( 35 ; 8 ; 30 ) , where relevant visual evidence is sparse, distributed across time. Static reasoning over uniformly sampled frames often causes major information loss and poor temporal localization, while also being computationally costly for long videos with hundreds of frames. A promising avenue for improving long-video understanding lies in the concept of interleaved multimodal reasoning, which involves combining visual perception with language reasoning in a dynamic and adaptive manner 50 ; 32 ; 12 . Inspired by successful techniques in the image domain, such as those seen in OpenAI’s o3 model ( 23 ) , concurrent work attempts to extend such interleaved reasoning to videos, proposing the notion of thinking with videos ( 21 ; 46 ; 9 ) . However, these approaches face key limitations: (1) they often rely on proprietary models such as Gemini-Pro ( 5 ) to construct agentic video-understanding data, which requires extensive prompt engineering and multi-stage filtering; Moreover, although Gemini-Pro can produce interleaved video-reasoning traces, it still lacks the emergent video-interleaved reasoning capability observed in o3’s image reasoning ( 23 ) ; and (2) they primarily perform single temporal zoom operations without a temporal retrieval mechanism, making them inefficient at identifying key temporal intervals. In contrast, open-source LLMs such as Qwen3 ( 38 ) and DeepSeek ( 20 ) exhibit strong tool-augmented reasoning capabilities as language agents for long-form video understanding. They employ various tools, with VideoLLMs generating intermediate frame captions as tools to support video reasoning ( 31 ; 6 ; 43 ; 34 ) . However, in such frameworks, the VideoLLM serves merely as a passive captioning module, while the LLM itself cannot directly perceive visual information. Motivated by these insights, we propose VideoThinker , an agentic VideoLLM trained entirely on synthetic tool-interaction trajectories. The core idea is to first convert videos into rich textual captions and then leverage a powerful agentic LLM to generate multi-step tool-use sequences within this caption space. To enable effective long-video reasoning, we design two complementary agentic tools: (1) Temporal Retrieval , which identifies candidate temporal intervals that may contain relevant information by leveraging audio transcripts (as subtitles), scene descriptions, and subtitle-based summaries; and (2) Temporal Zoom , which inspects intervals at a finer granularity through more detailed subtitles or frames. By leveraging these tools together with the LLM’s tool-augmented reasoning capability, we construct multi-turn tool-interaction trajectories. When the LLM invokes the temporal-zoom tool, the VideoLLM generates intermediate textual captions, which are subsequently replaced with actual video frames to form fully video-interleaved reasoning data. These trajectories are then used to fine-tune the VideoLLM, enabling it to actively retrieve and perceive key frames during reasoning, effectively bridging the gap from text- and image-based reasoning to true video reasoning. Furthermore, VideoThinker incorporates a confidence-gated tool controller, achieving strong performance gains: +6.8% on MLVU and +10.6% on LVBench over vanilla VideoLLMs, and +3.9% and +3.5% over caption-only LLM agents equipped with our tools. Our main contributions are summarized as follows: • We construct a high-quality agentic video–tool reasoning dataset guided by LLM-based tool reasoning. This synthetic data captures diverse temporal reasoning and tool-use behaviors essential for long video understanding. • We equip VideoThinker with dynamic reasoning, adaptive temporal exploration, and multi-step tool use via Temporal Retrieval and Temporal Zoom for efficient long-video understanding. • Extensive experiments demonstrate that VideoThinker outperforms strong VideoLLM and LLM-agent baselines across long-video benchmarks, validating the effectiveness of tool-augmented synthetic data and adaptive retrieval–zoom reasoning. 2 Related Work Figure 2 : VideoThinker integrates retrieval and zoom tools for multi-turn reasoning. LLMs use caption_zoom to generate reasoning data from videos, which is later replaced by frame_zoom and video frames to build agentic video understanding CoTs. 2.1 Agentic LLMs for Video Understanding Recent advances in large language models (LLMs), particularly their reasoning and planning capabilities, have spurred the development of agents 40 ; 10 . By leveraging external tools 24 ; 44 , LLMs can acquire information, plan, and execute actions in complex environments, bridging the gap between general-purpose language models and real-world applications. Extending this paradigm to video understanding, recent works integrate LLM reasoning with agentic tools to tackle complex video tasks without costly training. However, applying such agents to video understanding poses challenges, as many existing tools do not generalize well across temporal sequences. Several works have proposed video-specific adaptations of the LLM-agent paradigm 31 ; 6 ; 43 ; 15 . LifeLongMemory 33 constructs a text-based episodic memory from video narrations, enabling LLMs to reason and retrieve relevant information for downstream tasks. DoraemonGPT 39 introduces a MCTS-based prompting strategy, combining tool use with structured memory for video reasoning. VideoAgent 31 frames video understanding as a sequential decision-making process, where the agent decides whether to gather more information or produce an answer, mimicking human video interpretation strategies. VideoTree 34 enhances efficiency and interpretability by adaptively extracting key visual information in a coarse-to-fine manner and organizing it in a human-readable tree for LLM reasoning. Related approaches, such as DVD 47 and VideoExplorer 42 , employ agentic search strategies over segmented video clips to tackle long-form VideoQA. Together, these works demonstrate the potential of agentic LLMs for structured, tool-augmented reasoning in complex video understanding tasks. A key limitation of these approaches is that they

Read the complete text