Senses Wide Shut: A Representation–Action Gap in Omnimodal LLMs

Nguyen Quang Trung, Yiming Gao, Fanyi Pu, Kaichen Zhang, Shuo Sun, Ziwei Liu, Nanyang Technological University · 2026

Read Senses Wide Shut: A Representation–Action Gap in Omnimodal LLMs by Nguyen Quang Trung, Yiming Gao, Fanyi Pu, Kaichen Zhang, Shuo Sun, Ziwei Liu, Nanyang Technological University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13737.txt.

Introduction

Trung Nguyen Quang and five colleagues offer IMAVB, a curated 500-clip test set of long-form movies testing premise-sense conflict spotting in all-sense word models. Across eight open-source all-sense models and Gemini 3.1 Pro, hidden states well encode premise-sense clashes even when models almost never reject false claims. Models show two failure modes: under-refusal, answering tricky questions as if false claims were true. Over-refusal means saying no to standard questions too and losing grasp hits.

Catalog record

CREATORNguyen Quang Trung, Yiming Gao, Fanyi Pu, Kaichen Zhang, Shuo Sun, Ziwei Liu, Nanyang Technological University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112818
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Senses Wide Shut: A Representation–Action Gap in Omnimodal LLMs Nguyen Quang Trung, Yiming Gao, Fanyi Pu, Kaichen Zhang, Shuo Sun, Ziwei Liu, Nanyang Technological University Abstract When an omnimodal large language model accepts a question whose textual premise contradicts what it actually sees or hears, does the failure lie in perception or in action? Recent omnimodal models are positioned as perception-grounded agents that jointly process video, audio, and text, yet a basic form of grounding remains untested: catching a textual claim that conflicts with the model’s own sensory input. We introduce IMAVB, a curated 500-clip benchmark of long-form movies with a 2 × 2 2{\times}2 design crossing target modality (vision, audio) and premise condition (standard, misleading), which lets us measure conflict detection separately from ordinary multimodal comprehension. Across eight open-source omnimodal LLMs and Gemini 3.1 Pro, we document a Representation–Action Gap : hidden states reliably encode premise–perception mismatches even when the same models almost never reject the false claim in their outputs. Behaviorally, models fall into two failure modes: under-rejection , in which they answer misleading questions as if the false premise were true; and over-rejection , in which they reject more often but also reject standard questions, sacrificing ordinary comprehension accuracy. The gap is modality-asymmetric (audio grounding underperforms vision) and prompt-resistant across seven variants. As an initial diagnostic intervention, a probe-guided logit adjustment (PGLA) re-injects the encoded mismatch signal into decoding and consistently improves rejection behavior. Together, these results suggest the bottleneck for omnimodal grounding lies in translation, not perception. Figure 1 : Overview of the Representation–Action Gap on IMAVB. 1 Introduction Omnimodal large language models (LLMs) [ 51 , 17 , 59 ] are increasingly positioned as perception-grounded agents: they jointly process video, audio, and text, and are expected to understand textual claims against their own sensory input. Yet a basic form of grounding remains untested. If a question embeds an incorrect claim about what the model sees or hears (for example, the video shows a character in a maroon shirt but the question refers to a “blue” one), can the model catch the mismatch, or does it blindly trust the text? Silent compliance with false claims about the environment is a failure mode that cooperative benchmarks cannot surface, and it is central to any deployment that requires grounding against reality. The question behind this failure is sharper: when a model accepts such a false premise, is the failure one of perception (the mismatch was never detected) or one of action (a detected mismatch failed to propagate to the output)? Recent interpretability work on text-only LLMs has shown that models routinely encode information they do not express: internal states distinguish truthful from untruthful statements [ 1 , 6 , 37 ] , linearly encode refusal and honesty directions [ 73 , 29 ] , and predict answer correctness before generation [ 41 ] . Concurrent work extends this internal–external dissociation to vision-language models (VLMs): VLMs linearly encode visual evidence even when their answers ignore it [ 39 , 25 , 54 ] . Whether this dissociation extends to omnimodal models that jointly process video, audio, and text is unknown; whether it is modality-asymmetric across vision and audio is unknown; and whether it resists prompt-level interventions and remains actionable under a lightweight inference-time correction has not been tested. A parallel literature on false-premise questions [ 63 , 26 ] , sycophancy [ 43 , 50 , 57 , 31 , 67 ] , and honesty [ 35 , 72 ] shows that models often comply with user-asserted claims even when they have evidence to refuse, but this line of work has not yet connected behavioral sycophancy to its internal-state signature in the cross-modal perceptual setting. Existing cross-modal benchmarks cannot bridge the two: they either remove one modality [ 30 , 38 , 48 , 60 ] , assume cooperative premises [ 16 , 33 , 69 , 20 ] , or prompt the model to verify explicitly [ 52 , 65 ] . A detailed review is in Appendix K . To ask the perception-vs-action question, we construct IMAVB, an evaluation harness of 500 long-form movie clips arranged in a 2{\times}2 design over modality (vision, audio) and premise (standard, misleading). The video and audio content remain unaltered; only the textual question varies. Misleading variants copy the standard query but swap exactly one premise detail (e.g., “maroon shirt” \rightarrow “blue shirt”), expecting the model to reject via option E or F. Across eight open-source omnimodal LLMs and Gemini 3.1 Pro, we find a Representation–Action Gap . Behaviorally, seven of eight open-source models under-reject : they handle 40–75% of standard questions but reject misleading ones in only \leq 16.2% of vision and \leq 6.6% of audio cases, with four scoring 0% on audio under fixed option order; Qwen3-Omni and Gemini 3.1 Pro instead over-reject (72.8% and 94.0% on vision), trading 15–25pp of standard accuracy. Internally, however, linear probes recover the misleading premise from hidden states at up to 86%, with a 3–7pp residualized margin above text-only baselines on vision: the signal is linearly separable yet does not propagate to the output. The gap survives seven prompt variants and stratification by video length, and a probe-guided logit adjustment (PGLA) that re-injects the encoded signal at the output yields a +15.0pp gain across all eight open-source models, providing evidence that the signal is sufficient to shift behavior. Contributions. Our contributions are as follows: • We introduce IMAVB, a 500-clip benchmark of long-form movies in a 2{\times}2 design (§ 2 ) that combines intact video+audio stimuli, implicit (rather than telegraphed) false premises, and surgical modality targeting; this allows vision-grounded and audio-grounded failures to be measured separately while every modality remains present and competing, a configuration no prior omni benchmark provides. • We document a Representation–Action Gap that extends the text-only internal–external dissociation to audio and joint video+audio (omnimodal) grounding, and characterize it as modality-asymmetric (audio-grounded probes underperform vision-grounded ones, with a larger behavioral gap for audio misleading), present across all eight open-source architectures and on Gemini 3.1 Pro with heterogeneous underlying mechanism (§ 4.2 , § 4.3 ), and prompt-resistant across seven variants. • We propose a probe-guided logit adjustment (PGLA) that yields a +15.0pp mean improvement across eight models, providing diagnostic evidence that the encoded signal is sufficient to improve rejection when re-injected at the output. 2 IMAVB: An Omni Benchmark for Perception–Action Dissociation Asking whether an omnimodal failure arises from perception or from action requires an evaluation harness with five properties that no existing benchmark provides simultaneously: (i) all modalities must remain intact , because removing a modality prevents us from measuring cross-modal competition [ 30 , 38 , 48 , 60 ] ; (ii) false premises must be implicit , embedded as background assumptions rather than posed as explicit verification prompts that telegraph the test [ 52 ] ; (iii) modality targeting must be surgical , so vision-grounded and audio-grounded failures can be measured separately; (iv) stimuli must extend over time , supporting cross-modal tracking rather than instantaneous perception [ 16 , 33 , 69 , 20 ] ; and (v) symmetric standard controls must exist , permitting internal-gap measurement against the same stimuli. IMAVB is an omni benchmark design that

Read the complete text