Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning

Sara Ghazanfari, Francesco Croce, Nicolas Flammarion, Prashanth Krishnamurthy, Farshad Khorrami, Siddharth Garg · 2025

Read Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning by Sara Ghazanfari, Francesco Croce, Nicolas Flammarion, Prashanth Krishnamurthy, Farshad Khorrami, Siddharth Garg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00318.txt.

Introduction

Sara Ghazanfari and five co-authors teach video models to reason with explicit pointers to the frames they use, in a 2025 paper. Language models answer better when they first write out their reasoning. For video, earlier systems either stitched together heavy multi-step pipelines to pick frames, or reasoned loosely with weak ties to the timeline. The authors propose a single-stage method where the model's reasoning cites the exact frames behind each claim.

The approach, called Chain-of-Frames, keeps inference in one pass and stays self-contained. Naming frames inside the reasoning cuts the temporal mix-ups that plague video question answering. Better grounding matters for uses such as lecture search, surveillance review, and sports analysis, where an answer must point to the moment that proves it.

Catalog record

CREATORSara Ghazanfari, Francesco Croce, Nicolas Flammarion, Prashanth Krishnamurthy, Farshad Khorrami, Siddharth Garg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00318

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning Sara Ghazanfari, Francesco Croce, Nicolas Flammarion, Prashanth Krishnamurthy, Farshad Khorrami, Siddharth Garg Abstract Recent work has shown that eliciting Large Language Models (LLMs) to generate reasoning traces in natural language before answering the user’s request can significantly improve their performance across tasks. This approach has been extended to multimodal LLMs, where the models can produce chain-of-thoughts (CoT) about the content of input images and videos. In this work, we propose to obtain video LLMs whose reasoning steps are grounded in, and explicitly refer to, the relevant video frames. For this, we first create CoF-Data , a large dataset of diverse questions, answers, and corresponding frame-grounded reasoning traces about both natural and synthetic videos, spanning various topics and tasks. Then, we fine-tune existing video LLMs on this chain-of-frames (CoF) data. Our approach is simple and self-contained, and, unlike existing approaches for video CoT, does not require auxiliary networks to select or caption relevant frames. We show that our models based on CoF are able to generate chain-of-thoughts that accurately refer to the key frames to answer the given question. This, in turn, leads to improved performance across multiple video understanding benchmarks, for example, surpassing leading video LLMs on Video-MME , MVBench , and VSI-Bench , and notably reducing the hallucination rate. Code available at github.com/SaraGhazanfari/CoF . 1 Introduction Large Language Models (LLMs) are able to perform step-by-step reasoning, widely known as chain-of-thoughts (CoT) ( Wei et al. 2022 ; Kojima et al. 2022 ) . This capability has been implicitly integrated into state-of-the-art systems such as OpenAI’s o1/o3 models ( OpenAI 2024 ) and DeepSeek R1 ( Guo et al. 2025 ) , contributing to their remarkable performance and improving the interpretability of their internal functioning. CoT reasoning has been also extended to multimodal LLMs ( Hu et al. 2024 ; Wu and Xie 2023 ; Shao et al. 2024 ) : this presents new challenges compared to language-only domains, as the models need to attend to inputs from different modalities, and reason about both their individual content and how they are connected ( Awal et al. 2023 ; Kil et al. 2024 ; Sun et al. 2025 ) . Recent work has also begun to explore the integration of CoT into video understanding tasks, where the input to a multimodal LLM consists of a sequence of images (the frames of the video) along with a text prompt. This makes reasoning on videos particularly complex, as the model needs to capture the semantics of the text prompt, understand temporal and causal relationships between frames, and reason about the video in its entirety. Existing approaches rely on complex inference frameworks with architecture modifications ( Fei et al. 2024 ) or auxiliary networks ( Han et al. 2024 ) at evaluation time to integrate reasoning into video LLMs. This makes using these models both more computationally expensive and less general (as they are specialized to some tasks), and deviates from the natural CoT prompting successfully applied to standard LLMs. Another limitation of current methods is that collecting reasoning traces, needed for training, involves complex and costly procedures, which may limit how much the size of training datasets can be scaled in practice. For example, Wang et al. 2024b iteratively refines the samples via annotations from both LLMs and human experts, Han et al. 2024 leverages several auxiliary models, Fei et al. 2024 uses additional spatial-temporal scene graph data. Finally, video LLMs do not currently provide explicit connections between particular segments of the video and their reasoning, which would better ground the chain-of-thought. To remedy these limitations, in this work we propose chain-of-frames (CoF), a new frame-aware chain-of-thought reasoning approach for video LLMs that integrates temporal information directly into the CoT structure (see Fig. 1 ). This enables the model to identify and refer to the most relevant frames while answering questions, in contrast to prior works that treat frame selection and reasoning as separate stages. Chain-of-frames is a simple and natural adaptation of the CoT paradigm in NLP to video understanding that does not require the auxiliary networks or complex inference frameworks of existing methods. Moreover, we propose an efficient data generation pipeline that allows us to collect a large dataset of CoF examples, named CoF-Data . To achieve this, a key element consists of leveraging a synthetic video dataset ( Yi et al. 2020 ) to extract a large and diverse set of reasoning traces at virtually no cost. Then, we fine-tune two recent open-source video LLMs, InternVL2.5-4B ( Chen et al. 2024 ) and InternVL3-8B ( Zhu et al. 2025 ) , on our CoF-Data . In an extensive evaluation on five established benchmarks, we show that our CoF models significantly outperform the original ones, with and without naive CoT prompting ( Wei et al. 2022 ) (see Fig. 1 ). Moreover, they are competitive with or better than state-of-the-art (SOTA) open- and closed-source models (Table 2 ). Notably, our CoF-InternVL3-8B achieves higher accuracy than the best model reported on Video-MME , MVBench , and VSI-Bench leaderboards, surpassing even proprietary models like GPT-4o and Gemini 1.5 Pro. These results also demonstrate that video LLMs can learn the ability to produce frame-aware reasoning from a limited number of (synthetic) samples and generalize the behavior to unseen tasks. Thus, our work provides a simple and inexpensive approach to adapt the CoT paradigm for video LLMs that improves both performance and interpretability. (a) A CoF reasoning trace generated by our CoF-4B model. (b) CoF models vs the baseline models. Figure 1 : (a) Chain-of-frames reasoning generated by our CoF-InternVL2.5-4B model: it includes the key frames to answer the question (from Video-MME ). (b) Comparison of accuracy across multiple video understanding benchmarks between baseline models (InternVL2.5-4B and InternVL3-8B) and their CoF-enhanced counterparts: our models consistently outperform the baselines. Figure 2 : CoF-Data . We show examples of our training data with chain-of-frames reasoning, including video, question, answer and reasoning trace. We include samples from CoF-Data real (real video, top row) and CoF-Data synth (synthetic video, bottom), created as described in Sec. 3.3 . 2 Related Work Multimodal LLMs for videos. Multimodal Large Language Models ( Chen et al. 2023b ; Li et al. 2024b ; Ghazanfari et al. 2024 ) have made substantial progress in integrating visual and textual modalities, enabling them to perform complex reasoning and achieve deep understanding across various data types, including videos ( Chen et al. 2023b ; Xue et al. 2024 ; Li et al. 2023a ; Zhang et al. 2024b ; Zhang et al. 2024c ; Li et al. 2024a ; Bai et al. 2024 ; Zhu et al. 2025 ) . Among recent advancements, InternVL2.5 ( Chen et al. 2024 ) , InternVL3 ( Zhu et al. 2025 ) , LLaVA-NeXT-Video ( Zhang et al. 2024c ) , and Qwen2-VL ( Wang et al. 2024a ) stand out for their strong video understanding capabilities. Both InternVL and LLaVA-NeXT-Video process videos in an image-text interleaved format, aligning sequences of video frames with language to form a unified multimodal stream. In addition to these open-source models, closed-source systems such as GPT-4o ( Achiam et al. 2023 ) and Gemini-1.5 (

Read the complete text