From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA
Sena Korkut, María Alejandra Bravo Sarmiento, Sanghwan Kim, Zeynep Akata · 2026
Read From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA by Sena Korkut, María Alejandra Bravo Sarmiento, Sanghwan Kim, Zeynep Akata (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30220.txt.
Introduction
Sena Korkut, María Bravo Sarmiento, Sanghwan Kim, and Zeynep Akata audit traffic video question answering for a quiet failure: models answering without looking. Correct answers ought to rest on scene-specific visual evidence. Models may instead exploit textual shortcuts and language priors. High benchmark accuracy can hide that collapse completely. The authors measure visual dependence directly across four benchmarks.
Several recent open-weight vision-language models do suspiciously well blind. On one benchmark, removing video actually improves accuracy consistently. Adding more frames makes performance worse still. Two diagnostics quantify the problem at dataset level. Blind Gap measures above-chance performance from text alone. Visual Gain measures what video actually adds when present. An instance-level Shortcut Score then flags shortcut-prone questions. Filtering by that score yields subsets with less bias. Grounded evaluation, they argue, matters as much as accuracy.
Catalog record
| CREATOR | Sena Korkut, María Alejandra Bravo Sarmiento, Sanghwan Kim, Zeynep Akata |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA Sena Korkut, María Alejandra Bravo Sarmiento, Sanghwan Kim, Zeynep Akata Abstract High benchmark accuracy does not guarantee genuine use of visual evidence. We study this problem in traffic accident Video Question Answering (VideoQA), where correct answers should depend on scene-specific visual evidence but may instead be inferred from textual shortcuts. Through an audit of four public benchmarks, we find that several recent open-weight Vision-Language Models (VLMs) perform competitively, and sometimes better, without video input. On the MM-AU benchmark, removing video consistently improves accuracy, and adding more frames further degrades performance. To quantify visual dependence, we introduce two dataset-level diagnostics: Blind Gap, measuring above-chance text-only performance, and Visual Gain, measuring the marginal benefit of adding video. We further propose an instance-level Shortcut Score that combines text-only confidence with visual necessity signals, enabling continuous, training-free filtering of shortcut-prone questions. The resulting subsets reduce shortcut bias and improve visual grounding. Our findings reveal large differences in grounding quality across benchmarks and show that visually grounded evaluation, not just high accuracy, is essential in safety-critical VideoQA. Figure 1 : Diagnostic space of benchmark-level modality collapse. Each benchmark is positioned by its Blind Gap and Visual Gain, with circle size proportional to multimodal accuracy ( Acc_{V} ). Our shortcut-aware filtering shifts MM-AU and VRU-Accident toward lower Blind Gap and higher Visual Gain, producing more visually grounded evaluation subsets (G-MM-AU and G-VRU-Accident). 1 Introduction Benchmarks guide research priorities and serve as the primary basis for measuring progress. Yet benchmark gains do not always reflect genuine advances in the intended capability. In multimodal learning, models may answer correctly while ignoring visual input and relying instead on language priors or dataset-specific shortcuts ( 14 ; 1 ; 8 ) . This is especially concerning in traffic accident VideoQA, where correct answers should depend on scene-specific evidence such as interacting vehicles, pedestrians, temporal dynamics, and causal structure. We study this failure mode as modality collapse : the visual modality contributes little or no useful information despite the task nominally requiring visual reasoning ( 23 ; 9 ) . Across several traffic accident benchmarks, recent open-weight VLMs perform competitively, and sometimes better, without video input. On MM-AU, removing video consistently improves accuracy, and adding more frames further degrades performance. In contrast, SUTD-TrafficQA and VRU-Accident show measurable gains from video input. These results suggest that benchmark accuracy alone cannot distinguish genuine visual reasoning from shortcut exploitation. To evaluate visual dependence, we introduce a lightweight diagnostic framework based on two dataset-level metrics: Blind Gap , which measures how far text-only accuracy exceeds chance, and Visual Gain , which measures the benefit of adding video input. We further introduce a continuous instance-level Shortcut Score that combines confidence-weighted text-only evidence with a visual necessity signal, enabling shortcut-aware filtering without auxiliary training. We audit four public traffic accident VideoQA benchmarks: MM-AU, VRU-Accident, SUTD-TrafficQA, and AccidentBench. As shown in Figure 1 , the benchmarks occupy very different regions in the Blind Gap–Visual Gain space despite comparable accuracy, showing that benchmark accuracy alone does not reflect visual grounding quality. MM-AU lies in the shortcut-prone region, with high Blind Gap and negative Visual Gain, while TrafficQA shows lower Blind Gap and consistently positive Visual Gain. Shortcut-aware filtering shifts MM-AU and VRU-Accident toward the visually grounded region by reducing text-only solvability and increasing the utility of video input. These findings highlight that, in safety-critical VideoQA, systems must succeed because they use visual evidence, not because they exploit annotation artifacts. 2 Related Work VideoQA benchmarks have expanded multimodal evaluation to temporal and causal reasoning, yet benchmark accuracy alone cannot verify genuine visual grounding ( 6 ; 31 ) . VLMs inherit strong language priors from pretraining that persist even as visual capacity scales, pointing to a utilization rather than capacity bottleneck ( 13 ; 20 ) . Shortcut learning ( 12 ) in VQA has motivated binary filtering approaches ( 3 ; 31 ) and dataset-level auditing metrics ( 6 ; 29 ) , but neither identifies shortcut-prone questions at the instance level with a continuous score. Our work provides the first such audit along with a filtering framework across traffic accident VideoQA benchmarks. See Appendix A for a full discussion. 3 Diagnosing and Filtering Modality Collapse A VideoQA benchmark should require visual evidence to answer correctly. We define a question as visually grounded if video input improves prediction toward the correct answer, and shortcut-prone if it can be solved from text alone. We diagnose this at both the dataset and question levels to construct grounded evaluation subsets. 3.1 Stage 1: Dataset-Level Diagnostics We evaluate models in two settings: blind , where only the question and answer options are provided, and video , where uniformly sampled frames are added. The Blind Gap (BG) measures how far text-only accuracy exceeds random chance: \text{BlindGap}(\mathcal{D})=\text{Acc}_{\text{blind}}-\text{Acc}_{\text{random}}, (1) where \text{Acc}_{\text{random}}=1/K for a K -option question. The Visual Gain (VG) measures the accuracy change from adding video: \text{VisualGain}(\mathcal{D})=\text{Acc}_{\text{video}}-\text{Acc}_{\text{blind}}. (2) A high BG indicates text-only solvability, while positive VG indicates useful visual evidence. Together, high BG with low or negative VG signals shortcut exploitation, whereas low BG with positive VG indicates stronger visual grounding. 3.2 Stage 2: Question-Level Shortcut Diagnostics Dataset-level averages do not reveal which questions drive shortcut behavior. We therefore define a continuous per-question Shortcut Score S(q) from textual evidence and visual necessity. The Textual Evidence Score T(q) measures whether text-only LLMs answer correctly and confidently. For each model m\in\mathcal{M}_{\text{LLM}} , let H_{m}(q)=-\sum_{k=1}^{K}p_{m,k}(q)\log p_{m,k}(q) be the entropy of the answer distribution and H_{\text{norm},m}(q)=H_{m}(q)/\ln K\in[0,1] . Only correct blind predictions contribute: T(q)=\frac{1}{|\mathcal{M}_{\text{LLM}}|}\sum_{m\in\mathcal{M}_{\text{LLM}}}[\text{Correct}_{m}(q)]\cdot\bigl(1-H_{\text{norm},m}(q)\bigr). (3) High T(q) indicates a reliable linguistic shortcut. The Visual Necessity Score V(q) measures whether adding video increases the probability assigned to the correct answer. For each VLM, we compute: \Delta_{m}(q)=P_{m}(y^{\star}\mid\text{video},q)-P_{m}(y^{\star}\mid\text{blind},q), (4) and average the normalized gain across models: V(q)=\frac{1}{|\mathcal{M}_{\text{VLM}}|}\sum_{m\in\mathcal{M}_{\text{VLM}}}\frac{\Delta_{m}(q)}{1-1/K}. (5) Here, y^{\star} is the correct answer and K is the number of answer options. Positive V(q) indicates that video shifts probability toward the correct answer; near-zero or negative V(q) indicates that visual input is unnecessary or harmful. Table 1 : Global diagnosis of modality collapse. We report Blind Accuracy ( Acc_{B} ), Multimodal Accuracy ( Acc_{V} ), Blind Gap (BG \downarrow ), and Visual Gain (VG \uparrow ) in %. Text-only LLMs have no video accuracy by design. MM-AU TrafficQA Acc.Bench VRU-Acc. Model Acc_{B} Acc_{V} BG\downarrow VG\uparrow Acc_{B} Acc_{V} BG\downarrow VG\uparrow Acc_{B} Acc_{V} BG\downarrow VG\uparrow Acc_{B} Acc_{V} BG\downarrow VG\uparrow Average VLM 53.25 48.64 33.25 -4.62 30.89 42.71 5.89 11.83 29.97 33.05 4.73 3.09 46.91 62.61 21.91 15.70 InternVL 2.5 8B 52.14 50.35 32.14 -1.79 29.72 41.95 4.72 12.23 29.64 31.68 4.40 2.05 50.18 63.62 25.18 13.43 InternVL 3.5 8B 60.33 50.88 40.33 -9.46 33.62 45.67 8.62 12.06 30.43 37.76 5.19 7.33 42.95 62.85 17.95 19.90 Qwen 2.5 VL 7B 54.83 47.86 34.83 -6.97 31.52 43.53 6.52 12.01 30.50 35.18 5.26 4.69 44.40 60.92 19.40 16.52 LLaVA-OV 8B 45.70 45.45 25.70 -0.24 28.69 39.70 3.69 11.01 29.31 27.59 4.07 -1.72 50.10 63.03 25.10 12.93 Llama 3.1 8B 46.60 - 26.60 - 30.45 - 5.45 - 20.33 - -4.91 - 42.77 - 17.77 - Mistral 7B v0.1 33.92 - 13.92 - 28.52 - 3.52 - 26.60 - 1.36 - 31.18 - 6.18