Can Video Large Multimodal Models Think Like Doubters—or Double-Down: A Study on Defeasible Video Entailment
Yue Zhang, Jilei Sun, Yunhui Guo, Vibhav Gogate · 2025
Read Can Video Large Multimodal Models Think Like Doubters—or Double-Down: A Study on Defeasible Video Entailment by Yue Zhang, Jilei Sun, Yunhui Guo, Vibhav Gogate (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22385.txt.
Introduction
Yue Zhang, Jilei Sun, Yunhui Guo, and Vibhav Gogate test whether video models can revise a judgment when new facts arrive. Their 2025 paper calls this defeasible reasoning: in real life, fresh context can strengthen or weaken an earlier conclusion. Most video models commit to one reading and struggle to update it.
The team defines a new task with a video premise, a claim, and an update that shifts the claim's support. Models either classify the shift or write a fitting update. Their methods use counterfactual thinking and speech transcripts to reduce bias. A new benchmark data set, with updates labeled as strengtheners or weakeners, supports the tests.
Catalog record
| CREATOR | Yue Zhang, Jilei Sun, Yunhui Guo, Vibhav Gogate |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22385
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Video Large Multimodal Models Think Like Doubters—or Double-Down: A Study on Defeasible Video Entailment Yue Zhang, Jilei Sun, Yunhui Guo, Vibhav Gogate Abstract Video Large Multimodal Models (VLMMs) have made impressive strides in understanding video content, but they often struggle with abstract and adaptive reasoning—the ability to revise their interpretations when new information emerges. In reality, conclusions are rarely set in stone; additional context can strengthen or weaken an initial inference. To address this, we introduce Defeasible Video Entailment (DVidE), a new task that challenges models to think like doubters , constantly updating their reasoning based on evolving evidence. In DVidE, given a video premise and a textual hypothesis , models must determine whether a new update strengthens or weakens the hypothesis (classification version) or generate a coherent update that modifies the entailment relationship (generation version). For solving the classification task, we propose the Chain of Counterfactual Thought framework, utilizing counterfactual reasoning, ASR-enhanced video content, and rationale refinement to reduce inference bias. For the generation task, we develop a framework that combines ASR output with a Large Language Model (LLM) to produce coherent, contextually relevant updates aligned with the intended strengthener or weakener goals. Additionally, we introduce a novel benchmark dataset, with strengthener/weakener annotations and an LLM-based evaluation metric specifically designed for assessing generative performance. Experimental results demonstrate significant improvements, highlighting our proposed method in enhancing dynamic reasoning capabilities of VLMMs. 1 Introduction Video understanding has been a long-standing challenge in computer vision, encompassing tasks such as action recognition ( Materzynska et al. 2019 ; Ghadiyaram et al. 2019 ; Wang et al. 2019a ) , video captioning ( Lei et al. 2020 ; Wang et al. 2019b ; Han et al. 2023 ) , and video retrieval ( Hendricks et al. 2017 ; Gao et al. 2017 ) . Now, with the development of Video Large Multimodal Models (VLMMs) ( Maaz et al. 2024a ; Maaz et al. 2024b ) , video understanding tasks have become the key benchmark to test the visual ability of VLMMs. While existing VLMMs perform well in extracting perceptual details such as objects, activities, and temporal cues ( Li et al. 2024 ; Chen et al. 2023b ) , they often struggle with higher-level abstract reasoning needed to fully interpret complex scenes. Different from video understanding tasks, the video entailment ( Liu et al. 2020 ) task requires a model to assess whether a video premise supports or contradicts a textual hypothesis, typically assuming a fixed, definitive relationship between them. For example, in a sports video, if the hypothesis states, “The player scored a goal,” the task requires to determine whether the video confirms or refutes this claim. Figure 1: An example of defeasibility in video entailment. However, real-world reasoning is typically defeasible, meaning conclusions can be strengthened or weakened by new information. A video clip may initially support one interpretation, but additional cues, such as a line of dialogue from Automatic Speech Recognition (ASR) or a subtle shift in visual context, may alter that conclusion. As illustrated in Figure 1 , initially, Rachel’s frustration seems to stem from Gavin’s inappropriate behavior of staring at women. However, if new information reveals that he was actually on the phone with his mother discussing women’s body shapes, this strengthens the inference. Conversely, if he was just pretending Rachel was his secretary, this could suggest that her frustration stems from being misrepresented rather than from his behavior of staring at women. To address these challenges, we introduce Defeasible Video Entailment (DVidE), an extension of video entailment. Specifically, DVidE requires not only determining whether the initial relationship between a video and a hypothesis is entailment or contradiction, but also continuously revising that inference, either strengthening or weakening it, as new updates arrive. We explore this problem through two interrelated sub-tasks: (1) Classification Task: Given a video premise, a textual hypothesis, and a textual update, predict whether the update functions as a strengthener (increasing the plausibility of the hypothesis) or a weakener (making the hypothesis less likely). (2) Generation Task: Given a video premise, a hypothesis, and a goal (Weakener/Strengthener), generate a contextually grounded (Weakener/Strengthener) that meaningfully alters the original entailment relationship. In our preliminary experiments, we found that the performance of existing VLMMs was unsatisfactory for DVidE due to several key limitations: (1) Inability to Fully Capture Audio Information : Current VLMMs frequently struggle with audio-related tasks, failing to accurately comprehend and interpret audio semantics in the video. (2) Weak Visual Reasoning Ability : Biases in VLMM training can lead to limitations in visual reasoning, resulting in generated reasoning text that lacks diversity, particularly in out-of-distribution scenarios ( Wang et al. 2024 ) . (3) Poor Language Reasoning Ability : Vision-based fine-tuning can sometimes degrade an LLM’s original generative and reasoning capabilities ( Yan et al. 2024 ; Chai et al. 2024 ) . To address these limitations, we propose a Chain of Counterfactual Thought method to explore the diverse logical semantic relationships among the update, premise, and hypothesis, thereby reducing susceptibility to inherent inference bias in the classification task. Our approach consists of three key components. First, Counterfactual Thought-induced Rationale Generation explores diverse reasoning processes through counterfactual reasoning, enabling VLMMs to consider multiple perspectives and improve inference diversity. Second, ASR-enhanced Video Content Integration leverages an external ASR tool to supplement VLMMs’ perception of audio information, ensuring a more comprehensive understanding of video content. Third, Rationale Refinement and Selection utilizes the strong language reasoning capabilities of LLMs to refine the generated rationales, systematically removing inference errors introduced by VLMMs, and thereby improving final classification accuracy. Additionally, we incorporate ASR-enhanced Video Content Integration to refine video descriptions before prompting an LLM to refine the update based on the hypothesis, goal, and generated update, enhancing VLMMs’ generative capabilities for our generation task. To evaluate VLMMs, we employ annotators to construct the Defeasible Video Entailment benchmark. In addition to traditional generation metrics, we introduce a novel Large Language Model (LLM)-based evaluation metric specifically designed for our generation task. Experimental results demonstrate that both our classification method and generation method achieve significant improvements across multiple metrics. On the classification task, our approach achieves an accuracy improvement of 29.7% over the Video-ChatGPT model ( Maaz et al. 2024a ) ; On the generation task, our method increases the proportion of correctly generated updates from 7.50% to 60,63% on VideoLLaMA ( Cheng et al. 2024 ) . Our main contributions can be summarized as follows: (1) We introduce a benchmark for defeasible video entailment, augmenting existing video entailment datasets with strengthener/weakener annotations and extending video-language reasoning. (2) We develop a novel Chain of Counterfactual Thought classification framework that leverages counterfactual reasoning to explore the diverse entailment semantic relation. (3) We propose a generation framework that integrates the external ASR tool and LLM to produce coherent and contextually relevant updates while ensuring they align with the goal (Weakener/Strengthener). (4) We propose a novel LLM-based evaluation method for our generation task. Our experimental results demonstrate that our proposed methods significantly improve both classification and generation performance on DVidE. 2 Related Work Defeasible Inference Tasks. Defeasible inference extends traditional entailment ( Bos and Markert 2005 ; Dagan et al. 2005 ; MacCartney and Manning 2009 ;