TRoVe : Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models
Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari, Curtis Langlotz · 2025
Read TRoVe : Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models by Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari, Curtis Langlotz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01048.txt.
Introduction
Maya Varma and four coauthors present TRoVe in a 2025 paper on video-language models. These models handle temporal tasks, questions about how a scene changes across frames. Yet recent work suggests they often lean on static cues, like background or a single object's look, instead of real change.
TRoVe discovers error-inducing static feature biases in such models. It hunts for the frozen features that trick a model into wrong answers on temporal questions. Exposing those shortcuts matters for safety and science uses, where change over time is the whole point. A model that ignores motion should not be trusted with surveillance or sports video.
Catalog record
| CREATOR | Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari, Curtis Langlotz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TRoVe : Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari, Curtis Langlotz Abstract Vision-language models (VLMs) have made great strides in addressing temporal understanding tasks, which involve characterizing visual changes across a sequence of images. However, recent works have suggested that when making predictions, VLMs may rely on static feature biases , such as background or object features, rather than dynamic visual changes. Static feature biases are a type of shortcut and can contribute to systematic prediction errors on downstream tasks; as a result, identifying and characterizing error-inducing static feature biases is critical prior to real-world model deployment. Existing approaches for identifying such systematic failure modes in trained models (i) are typically designed for non-temporal settings and (ii) are challenging to evaluate in temporal settings due to the lack of quantitative evaluation frameworks. In this work, we address these challenges by introducing TRoVe , an automated approach for discovering error-inducing static feature biases learned by temporal VLMs. Given a trained VLM and an annotated validation dataset associated with a downstream classification task, TRoVe extracts candidate static features from the dataset and scores each feature by (i) the effect of the feature on classification errors as well as (ii) the extent to which the VLM relies on the feature when making predictions. In order to quantitatively evaluate TRoVe , we introduce an evaluation framework consisting of 101 trained temporal VLMs paired with ground-truth annotations for learned static feature biases. We use this framework to demonstrate that TRoVe can accurately identify error-inducing static feature biases in VLMs, achieving a 28.6% improvement over the closest baseline. Finally, we apply TRoVe to 7 off-the-shelf VLMs and 2 temporal understanding tasks, surfacing previously-unknown static feature biases and demonstrating that knowledge of learned biases can aid in improving model performance at test time. Our code is available at https://github.com/Stanford-AIMI/TRoVe . 1 Introduction Vision-language models (VLMs) capable of jointly processing visual and textual data have been shown to possess state-of-the-art reasoning abilities [ 35 , 18 , 52 , 39 , 24 , 27 , 43 ] . In particular, given an input sequence with multiple images collected across varying timepoints, temporal VLMs can effectively characterize visual changes over time, a capability known as temporal understanding [ 2 , 5 , 1 , 45 , 41 , 44 , 51 , 25 ] . For example, temporal VLMs can recognize human actions given a sequence of video frames [ 45 , 41 , 44 ] and characterize disease progression given longitudinal medical images [ 2 , 5 , 1 ] . Models designed to perform temporal understanding tasks often demonstrate high overall performance; however, recent works have demonstrated that such models may be affected by static feature biases , a phenomenon where models utilize static patterns (e.g. image background or a particular object in the scene) as shortcuts when making predictions rather than analyzing dynamic visual changes occurring across the image sequence [ 23 , 3 , 42 , 9 , 22 ] . As an illustrative example, consider videos from an activity recognition dataset with the class label “ climbing tree ”, which depict people climbing up or down trees (Figure 1 ). A temporal VLM is tasked with accepting a sequence of video frames as input and then classifying the action being performed by the person in the scene. In this scenario, a VLM that relies on static feature biases may base predictions for the class label “ climbing tree " solely on the presence of trees and foliage, rather than analyzing the true motion patterns associated with a person climbing a tree. At inference time, the performance of this VLM will depend heavily on whether the static feature is present; consequently, as illustrated in Figure 1 , the VLM is likely to make systematic prediction errors when classifying videos from other classes (e.g. “ swinging on something ") with prominently visible trees. Figure 1: Temporal models often rely on the presence of static feature biases, such as background features or objects in the scene, when making predictions. In this example, the VideoCLIP-XL model [ 41 ] makes systematic prediction errors on the class swinging on something when trees are present. Identifying learned static feature biases that contribute to systematic prediction errors is critical prior to real-world model deployment [ 32 ] . Traditional approaches for detecting such failure modes, which typically involve a combination of manual analysis and pixel-wise interpretability algorithms (e.g. GradCAM), require extensive human effort and are time-consuming to implement at scale, particularly as the length of the input sequence increases [ 36 , 49 ] . This suggests the need for automated approaches; however, performing automated identification of error-inducing static feature biases is challenging for the following two reasons. First, existing automated approaches for discovering systematic errors are designed for non-temporal (e.g. single image) settings [ 11 , 17 , 37 , 38 ] . Such approaches, which typically operate by analyzing model predictions on a labeled validation dataset and surfacing coherent groups of misclassified samples, are not adequate for discovering static feature biases in settings where each data sample consists of a sequence of multiple, temporally-linked images. Second, performing quantitative evaluations of automated approaches in the temporal setting is complicated by the fact that the ground-truth static feature biases of pretrained models are typically unknown; as a result, it is difficult to ascertain whether biases extracted by automated methods are indeed accurate. In this work, we address these challenges by introducing TRoVe , an automated approach for improving T emporal Ro bustness of V ision-Languag e models. Given a pretrained VLM, our goal is to discover learned static feature biases that contribute to systematic prediction errors on downstream temporal understanding tasks. Knowledge of such static feature biases (e.g. trees in the previously-discussed example) can enable a developer to better understand and address model failure modes prior to real-world deployment. To this end, TRoVe operates on a labeled validation dataset by first decomposing each input multi-image sequence into constituent images and grouping visually-similar images into clusters. Here, each cluster represents a particular feature occurring consistently throughout the dataset. We then introduce a scoring function that ranks each feature by (i) the effect of the feature on classification errors as well as (ii) the extent to which the VLM relies on the feature when making predictions. As output, TRoVe generates a list of identified static feature biases paired with affected class labels. In order to assess the utility of our approach, we design an evaluation framework consisting of 101 temporal VLMs trained on synthetic data. We pair each VLM with annotations for ground-truth error-inducing static feature biases, enabling rigorous quantitative analyses. Across this suite of models, TRoVe accurately discovers error-inducing static feature biases, achieving a 28.6% improvement over the closest baseline. We find that TRoVe operates effectively across a range of static feature bias types (background biases, object biases, and attribute biases) and input sequence lengths. Given the strong performance of TRoVe on synthetic experimental settings, we then extend TRoVe to real-world temporal VLMs. Across a suite of seven state-of-the-art VLMs and two temporal understanding tasks (activity recognition in videos and disease progression