ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validation
Tony Montes Fernando Lozano The Department of Electrical and Electronics Engineering Universidad de los Andes {t.montes, flozano}@uniandes.edu.co · 2025
Read ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validation by Tony Montes Fernando Lozano The Department of Electrical and Electronics Engineering Universidad de los Andes {t.montes, flozano}@uniandes.edu.co (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15928.txt.
Introduction
Tony Montes and Fernando Lozano present an agent that answers questions about videos without task training. The agent uses a large language model as its brain and splits hard questions into steps. It also tracks named objects in frames to ground its claims.
A vision tool with open word lists helps it find and follow those objects. It can cross check the times it cites, which lifts trust in its replies. Tests on three known sets set new top scores. The design points to more checked, less made up video answers in the future.
Catalog record
| CREATOR | Tony Montes Fernando Lozano The Department of Electrical and Electronics Engineering Universidad de los Andes {t.montes, flozano}@uniandes.edu.co |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validation Tony Montes Fernando Lozano The Department of Electrical and Electronics Engineering Universidad de los Andes {t.montes, flozano}@uniandes.edu.co Abstract Recent advancements in Video Question Answering (VideoQA) have introduced LLM-based agents, modular frameworks, and procedural solutions, yielding promising results. These systems use dynamic agents and memory-based mechanisms to break down complex tasks and refine answers. However, significant improvements remain in tracking objects for grounding over time and decision-making based on reasoning to better align object references with language model outputs as newer models get better at both tasks. This work presents an LLM-brained agent for zero-shot Video Question Answering (VideoQA) that combines a Chain-of-Thought framework with grounding reasoning alongside YOLO-World to enhance object tracking and alignment. This approach establishes a new state-of-the-art in VideoQA and Video Understanding, showing enhanced performance on NExT-QA, iVQA, and ActivityNet-QA benchmarks. Our framework also enables cross-checking of grounding timeframes, improving accuracy and providing valuable support for verification and increased output reliability across multiple video domains. The code is available here . Figure 1 : An overview of our ViQAgent framework . Through three main modules, we propose an agentic solution for the Video Question-Answering (VideoQA) task by taking advantage of most advanced VideoLLMs capabilities on first-sight zero-shot reasoning, timeframe captioning, and target identification ( M_{1} ) , and the open-vocabulary capabilities of YOLO-World to ground the given targets/objects in the video ( OG ) in specific parts of the video in between t_{0} and t_{f} ; to finally end with a Chain-of-Thoughts judgment and reasoning layer (M_{2}) that compares both the grounded context and grounded object detections to determine the confidence of the M_{1} answer. In case of discrepancy, the CoT judge defines a set of clarification questions in specific timeframes that go through the VideoLLM again for specific short-ended question-answering. Finally, a reasoning layer takes these answers and the original question to produce a grounded and more accurate answer. 1 Introduction In the present year, the evolution of large language models (LLMs) [ 26 , 32 , 11 ] and vision language models (VLMs) [ 19 , 49 , 15 , 21 , 14 , 2 , 1 ] has significantly advanced their ability in video understanding, particularly in the video question-answering (VideoQA) task, a significant challenge on computer vision, where the model is provided with a video and a related question [ 31 , 38 , 6 , 29 ] that they must answer as accurately as possible. These models are designed to analyze the visual and linguistic data of the video to generate answers based on semantics and dynamics. Despite recent progress, significant limitations still need to be addressed when addressing more complex videos, particularly those with dynamic context and extensive length. Current LLM-based solutions often struggle with reliably capturing content that is crucial for answering questions when scenes are complex or require a high level of contextual and sequential understanding. This highlights the need for more adaptable approaches that can respond to a broader range of video types and question complexities [ 17 , 25 , 40 , 20 ] . To address these challenges, agent-based strategies for VideoQA have recently emerged, employing modular reasoning blocks [ 23 , 28 ] , memory-based strategies [ 7 , 23 ] . Procedural approaches [ 4 , 28 ] , as well as employing foundational vision-language models as tools for solving complex tasks and augmenting the context [ 34 , 4 , 8 ] . This enables models to track relevant content more effectively over time, which results especially useful for structuring tasks and reasoning through multiple steps, enabling more accurate question answering even when handling diverse and dynamic content. A key approach in these frameworks is video grounding [ 6 , 39 , 23 , 8 ] , wherein specific video segments are identified as containing objects or events essential to answering the question. In this case, Grounding anchors target segments within the video, allowing the model to focus on relevant portions rather than attempting to analyze entire video sequences. These approaches are also increasingly incorporating Chain-of-Thought (CoT) reasoning. In this method, the model explicitly articulates step-by-step reasoning with purposes of evaluating the consistency [ 39 , 12 , 27 ] , planning the steps to solve a task [ 28 ] , or just bringing a more accurate and argued response [ 24 , 50 , 8 , 23 ] . CoT reasoning can enhance certainty in the model’s responses, improve interpretability by presenting intermediate reasoning that is understandable to humans, and validate model outputs against additional contextual information. Inspired by these advances, we propose ViQAgent. This framework combines the capabilities of VideoLLMs and state-of-the-art vision-language foundation models to create a structured agent for VideoQA. ViQAgent employs a VideoLLM [ 32 ] to identify the key objects ( i.e . , targets) in the video that are relevant to the question, generating an initial understanding of the main elements and dynamics in a video and a preliminary answer based on observed video content. This initial answer ( i.e . , first-sight response) includes a first attempt to ground it in time and provides a set of key timeframes with their corresponding captions in the video. In the next phase, ViQAgent utilizes YOLO-World [ 3 ] to perform object detection based on this list of targets. Here, YOLO-World tracks occurrences of these predefined targets across video frames, returning an accurate timeline of their appearances, a task that sometimes VLMs tend to struggle with, when compared against each other [ 18 ] . Unlike traditional object detectors with fixed categories, YOLO-World’s tracking is customized by the VideoLLM’s selected open-vocabulary targets, allowing it to focus detection on the objects that are most pertinent to solving the question. The timeline created in this step adds a layer of precision, extending the initial detection with frame-by-frame data for each target object and the count of detected objects. In the final phase, the judgment and reasoning layer compares the initial response plus its grounded context, as well as the comprehensive object-tracking data, to determine whether the response is confident or not. This reasoning layer applies Chain-of-Thought (CoT) reasoning to combine both inputs, carefully validating the initial reaction against the grounded data from YOLO-World to produce an answer that is both accurate and substantiated by clear visual evidence. In case of an unconfident answer, a set of expressly framed questions are re-validated through the VideoLLM to provide a final answer then. The CoT approach in this output stage reinforces interpretability by tracing logical steps in reaching the answer and strengthens answer reliability by cross-checking the data from both sources. Our approach provides multiple advantages as a zero-shot solution, requiring no specialized task-based fine-tuning and thus allowing easy adaptation to new scenarios and question types with just a dataset-specific subprompt. By combining the strengths of VideoLLMs for initial analysis and general video understanding and YOLO-World for open-vocabulary object detailed tracking, it benefits from the complementary capabilities of these models, handling a wide range of video-based questions with minimal configuration. Furthermore, the CoT mechanism contributes to a more nuanced and compelling cross-validation of the reasoning outputs. This grounding structure improves reliability