When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?
Qilang Ye, Wei Zeng, Meng Liu, Jie Zhang, Yupeng Hu, Zitong Yu, Yu Zhou · 2025
Read When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? by Qilang Ye, Wei Zeng, Meng Liu, Jie Zhang, Yupeng Hu, Zitong Yu, Yu Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10059.txt.
Introduction
Qilang Ye and six co-authors test whether multimodal language models can notice when a video's sound does not match its picture. They built AV-ConfuseBench, which alters an object's sound, for example muting a sounding object, then asks whether that sound is present. Leading models, including Qwen2.5-Omni and Gemini 2.5, largely failed, reasoning from what they saw rather than what they heard.
Their remedy, RL-CoMM, adds a large audio language model that reasons from sound alone as a reference. A step-wise reward then teaches the main model to improve its audio-visual reasoning against that reference, and a confidence optimization stage steadies the final answer. Across audio-visual question answering and hallucination tests, accuracy rose by roughly ten to thirty points over the baseline with limited training data.
Catalog record
| CREATOR | Qilang Ye, Wei Zeng, Meng Liu, Jie Zhang, Yupeng Hu, Zitong Yu, Yu Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10059
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? Qilang Ye, Wei Zeng, Meng Liu, Jie Zhang, Yupeng Hu, Zitong Yu, Yu Zhou Abstract Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an “Audio-Visual Confusion” scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs “ Is there a/an { muted-object } sound ”. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM , a R einforcement L earning-based Co llaborative M ulti- M LLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves the accuracy by 10 ∼ \sim 30% over the baseline model with limited training data. Follow: https://github.com/rikeilong/AVConfusion. Introduction We perceive things by gathering visual information from our eyes while constantly acquiring knowledge through our hearing. Recent advancements in Multimodal Large Language Models (MLLMs 1 1 1 In this paper, MLLMs refer to models that can process inputs containing both video and audio. ) ( 40 ; 1 ; 44 ) show a remarkable ability to understand real-world human language and generate continuum contexts. Furthermore, the wide range of complex visual and audio signals has facilitated the development of MLLMs such as Qwen2.5-Omni ( 35 ) and Gemini 2.5 Pro ( 26 ) . After fine-tuning with an extensive synchronized video and sound corpus, MLLMs develop a cognitive capacity for audio-visual understanding in areas including automatic speech recognition ( 23 ) , audio-visual captioning ( 43 ) , and general audio-visual processing ( 8 ) . Despite the excellent generative capabilities of MLLMs for generic scenarios, studies on Audio-Visual Hallucinations (AVH) ( 6 ) have revealed that MLLMs are unable to distinguish between the volume and pitch of two pieces of audio. Furthermore, MLLMs are susceptible to hearing fictitious sounds or perceiving imaginary visual objects in cross-modal understanding scenarios ( 31 ) . Figure 1: Examples of MLLMs confronting audio-visual confusion. In this paper, we further explore: if the given audio-visual information is asymmetric, \diamondsuit Can MLLMs discern confused objects that are visually present but audio-absent? We call this phenomenon “ Audio-Visual Confusion ”, which refers to the given audio-visual information being inherently asymmetric. As illustrated in Fig. 1 , we primarily focus on two settings: (a) Test whether MLLMs can recognize that a certain muted object is not making a sound . In normal cognition, vision and audio are bound, i.e., the presence of some object necessitates the presence of sound feedback. But what if given a damaged video that is visually intact but acoustically crippled, would MLLMs recognize it? For example, we input a “cello-muted” video and ask Baichuan-Omni-7B ( 11 ) , Gemini 2.5 Pro ( 26 ) , and Qwen2.5-Omni-7B ( 35 ) if there is a cello sound. All models fail to notice the missing audio but blindly believe in the visual information. Further, we explore (b) Test whether MLLMs can balance audio-visual information for reasoning . For example, we input an extremely unbalanced video (tampering with the music as bird sounds) to Qwen2.5-Omni. Then, we follow an entropy-based metric ( 4 ; 49 ) to quantify the uncertainty of the next token in each layer. As shown on the right of Fig. 1 , although Qwen2.5-Omni is able to distinguish bird sounds, its uncertainty scores remain relatively high in the latter layers, which means the model is still being biased towards hearing acoustic guitar and keyboard sounds when generating responses. To further investigate this phenomenon, we propose a mini-benchmark, AV-ConfuseBench , to assess the ability of MLLMs to distinguish audio-visual confusions. It consists of two settings: audio-muted (i.e., muting one of the multiple instruments all the time) and audio-modified (i.e., tampering with the soundtrack of the entire video to be completely out of sync with the theme). Audio-muted setting is tested by giving a query: “Is there a/an {muted-object} sound?” and models reply yes/no; audio-modified setting aims at evaluating the consistency of the generated content with the audio-visual ground truth. As shown in Table 1 , we find that: 1) None of the MLLMs, especially open-source models, yield the expected results in the audio-muted scenario, which means they struggle to discriminate non-existent audio tokens due to visual influences. 2) While Gemini 2.5 Pro with thinking mode achieves higher performance over the baseline, it remains difficult to think outside of the inherent internal knowledge in multimodal tasks containing video and audio (About 38.36% of the responses remain affected by visually guided thinking and output errors). 3) MLLMs appear to be insensitive to audio content, and the reasoning is dominated by the presented visual in the audio-modified scenario. This means the generated contexts mostly elaborate visual information. Motivated by this observation, we introduce RL-CoMM , a R einforcement L earning-based Co llaborative M ulti- M LLMs system that is built upon the Qwen2.5-Omni-3B foundation. RL-CoMM consists of two different stages: 1) Step-wise Reasoning Reward function (Step-RR) and 2) Answer-centered Confidence Optimization (Ans-CO). For the first stage, RL-CoMM introduces an external model, i.e., Large Audio Language Models (LALMs), as the reference model to supplement audio evidence. Then, we design Step-RR based on Group Relative Policy Optimization (GRPO), which uses the extra audio knowledge to incentivize the policy model for audio context reasoning and audio-visual correlation reasoning. For the second stage, Ans-CO can reduces the policy model’s uncertainty over its answer predictions to resolve uncertainties arising from potentially heterogeneous reasoning differences. Through extensive experiments on Audio-visual Question Answering (AVQA) tasks and AVH benchmarks, our proposed RL-CoMM improves accuracy by 10 \sim 30% over the base-LLM only with around 20% of total training samples. Model Audio-muted Audio-modified Acc. ( \uparrow ) Yes (%) A-Acc. ( \uparrow ) V-Acc. ( \uparrow ) Open-source omni-models Video-LLaMA2-7B 2.73 97.27 0.88 3.88 Baichuan-Omni-7B 5.47 94.53 1.12 4.07 Qwen2.5-Omni-7B 9.59 90.41 1.02 4.32 Close-source omni-models Gemini 2.5 Flash 28.76 71.24 2.24 4.78 Gemini 2.5 Pro 68.50 31.50 2.83 4.66 Random Choice 50.00 50.00 - - Table 1: Results of different MLLMs on AV-ConfuseBench. We evaluate various open/closed-source in two settings, where “Yes” is the proportion of answer yes among total responses, “A-Acc.”, “V-Acc.” refer to scoring audio and visual response accuracy on a 0-5 scale using GPT-4. Related Work Large Audio-Visual Language Models. Drawing inspiration from the remarkable ability of Large Language Models (LLMs) ( 36 ; 37 ; 32 ) to generate coherent language, studies have extended LLMs to other multimodal tasks, e.g., Audio-visual Question Answering (AVQA) ( 42 ) , and human-centric understanding tasks ( 22 ; 41 ) . Primarily,