A Video Is Not Worth a Thousand Words
Sam Pollard, Michael Wray · 2025
Read A Video Is Not Worth a Thousand Words by Sam Pollard, Michael Wray (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23253.txt.
Introduction
Sam Pollard and Michael Wray question whether video question answering benchmarks truly test video understanding, under the title that a video is not worth a thousand words. Benchmarks keep getting harder and models keep using longer contexts, often with large language model backbones. That raises a worry: models may lean on text and barely use the moving image.
The authors probe interactions between modalities and ask how to measure real progress. If a model answers from language priors alone, high scores mislead. Their critique pushes the field to design tests where watching the video is genuinely required.
Catalog record
| CREATOR | Sam Pollard, Michael Wray |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Video Is Not Worth a Thousand Words Sam Pollard, Michael Wray Abstract As we become increasingly dependent on vision language models (VLMs) to answer questions about the world around us, there is a significant amount of research devoted to increasing both the difficulty of video question answering (VQA) datasets, and the context lengths of the models that they evaluate. The reliance on large language models as backbones has lead to concerns about potential text dominance, and the exploration of interactions between modalities is underdeveloped. How do we measure whether we’re heading in the right direction, with the complexity that multi-modal models introduce? We propose a joint method of computing both feature attributions and modality scores based on Shapley values, where both the features and modalities are arbitrarily definable. Using these metrics, we compare 6 6 VLM models of varying context lengths on 4 4 representative datasets, focusing on multiple-choice VQA. In particular, we consider video frames and whole textual elements as equal features in the hierarchy, and the multiple-choice VQA task as an interaction between three modalities: video, question and answer. Our results demonstrate a dependence on text and show that the multiple-choice VQA task devolves into a model’s ability to ignore distractors. Code available at https://github.com/sjpollard/a-video-is-not-worth-a-thousand-words . 1 Introduction Since the advent of pre-trained large language models (LLMs) with strong reasoning capabilities, vision language models (VLMs) have rapidly become a catchall system for users desiring to interact with multi-modal models, primarily because they can be queried in a similar manner to humans. VLMs are frequently modelled as a paired vision encoder and a pre-trained LLM, where visual and text tokens are projected into the same input space. With larger and more powerful models, VLMs are now created for visual understanding, either specialising in video or additionally allowing multi-image input. These models place the onus to reason well on the LLM, assuming that so long as the features are well-aligned, the model will understand the relationship between the modalities. We explore this hypothesis by determining the degree that each modality feature contributes to a model’s response, and hence, whether modality preferences present themselves in current approaches to video question answering (VQA)—a common benchmark task for video language models. Our effort stems from an intuition that video is not being reliably integrated into VLMs, motivated by a variety of related work. Recently, Deng et al. 2025 showcased a blind reliance on text when modalities disagree with eachother. Using Shapley values ( Shapley 1997 ) , the modality preference of image/text tasks was investigated by Parcalabescu & Frank 2023 ; Parcalabescu & Frank 2025 , showing that VLMs are less self-consistent than LLMs. The separate, but related, problem of video redundancy has been investigated by works including Buch et al. 2022 , which aimed to determine “what can be understood from a single image”, by training a probe to select single frames capable of answering questions intended to be temporally complex. Similarly, Price & Damen 2020 showed that uni-modal video model performance for action recognition could be improved by removing “distractor” frames, calculated via Shapley values. Where previous works have used token level features for calculating attributions and accuracy-based heuristics for determining modality preference ( Parcalabescu & Frank 2023 ; Goldshmidt & Horovicz 2024 ) , we instead propose an attribution method for arbitrarily grouped features that can be used to calculate modality scores. We consider whole frames and text elements (words, numbers, etc.) as individual features, as these are the smallest unit of meaningful information from a human perspective and this incorporates any model specific encoding/tokenisation while reducing the complexity of the input. Conducting this analysis across a range of VQA datasets, we jointly investigate current capabilities of open source multi-modal models to properly integrate video and text. In summary, our contributions are as follows: (i) we expand upon feature attribution methods built on Shapley values, being the first to bring it to the video/text domain; (ii) propose a method of calulating modality scores independent of model accuracy and modality length; (iii) benchmark 6 VLM models on 4 datasets; (iv) determine that video as an entire modality is being under-utilised by models—frames are consistently undervalued compared to text and (v) demonstrate that video contribution (and dataset difficulty) can be increased simply by adding more multiple-choice answers. 2 Related Work Interpretability Early interpretability approaches focused on uni-modal models, e.g., vision: ( Fan et al. 2022 ; Koh et al. 2020 ; Wong et al. 2021 ; Smilkov et al. 2017 ) , audio ( Parekh et al. 2024 ; Mishra et al. 2017 ; Becker et al. 2018 ) , and language: ( Ribeiro et al. 2016a ; Bau et al. 2017 ; Chen et al. 2024a ; Shi et al. 2023 ; Xie et al. 2024b ) . Works focused on gradient based methods ( Sundararajan et al. 2017 ; Binder et al. 2016 ; Selvaraju et al. 2017 ) and attention-based methods concurrent with the rise of transformers ( Jain & Wallace 2019 ; Wiegreffe & Pinter 2019 ) . Others employed feature attribution methods ( Goldshmidt & Horovicz 2024 ; Price & Damen 2020 ) using Shapley values ( Shapley 1997 ) from game theory, which calculates the contribution of players within a co-op game. Shapley values have been employed as a black box approach, requiring only perturbations to the input and examining the output to interpret the model. Specifically, SHapley Additive exPlanation (SHAP) ( Lundberg & Lee 2017 ) , unified Shapley values such that given a prediction, each input feature is assigned an importance value. With the rise in popularity of vision language models, recent work has also focused on interpretability across multi-modal models ( Stan et al. 2024 ; Chen et al. 2024d ; Aflalo et al. 2022 ; Chefer et al. 2021 ; Liu et al. 2024 ; Ramesh & Koh 2022 ; Swamy et al. 2023 ) . DIME ( Lyu et al. 2022 ) —itself an extension of LIME ( Ribeiro et al. 2016b ) —interprets models via the pertubation of input features to affect the model outputs. It first disentangles the model into uni-modal and multi-modal contributions before generating intepretable visualisations via applying LIME. Similar to our work, MM-SHAP ( Parcalabescu & Frank 2023 ) extends SHAP for multiple modalities, in this case for image and text, to measure contributions of image patches and words in text to the final decision made by the model. PixelSHAP ( Goldshmidt 2025 ) , was proposed to instead model the contributions of feature groups of pixels related to the same concept within the input image, thus allowing for a finer granularity than attributions of image patches. Our work lies within the feature attribution paradigm, extending SHAP for the task of VQA enabling both frame attribution and modality dependence for video, question, and answer inputs. Modality Preference Concurrent with interpretability approaches, Huang et al. 2022 theorise that “During joint training, multiple modalities will compete with each other [and only a subset of modalities will be utilised] … with other modalities failing to be explored.” Many works have reached similar conclusions by exploring modality dominance. These works have focused on: pertubation of input modalities ( Park