Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation

2025

Read Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06803.txt.

Introduction

Xilin Jiang and three co-authors compare in 2025 how well large AI models recognize sounds by ear, by eye, and against humans. Audio models are billed as sound experts, yet nobody had lined them up against vision models and people on the same task: naming sound-making objects from audio, silent video, or sounded video.

A gap appeared between the audio and visual models that mirrored the human gap between hearing and seeing. The team then taught each model from the other, aiming transfers at the classes a guide model flagged as hard for the student. Teaching in both directions lifted scores, notably on the hard classes.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation Abstract Audio large language models (LLMs) are considered experts at recognizing sound objects, yet their performance relative to LLMs in other sensory modalities, such as visual or audio-visual LLMs, and to humans using their ears, eyes, or both remains unexplored. To investigate this, we systematically evaluate audio, visual, and audio-visual LLMs, specifically Qwen2-Audio, Qwen2-VL, and Qwen2.5-Omni, against humans in recognizing sound objects of different classes from audio-only, silent video, or sounded video inputs. We uncover a performance gap between Qwen2-Audio and Qwen2-VL that parallels the sensory discrepancy between human ears and eyes. To reduce this gap, we introduce a cross-modal distillation framework, where an LLM in one modality serves as the teacher and another as the student, with knowledge transfer in sound classes predicted as more challenging to the student by a heuristic model. Distillation in both directions, from Qwen2-VL to Qwen2-Audio and vice versa, leads to notable improvements, particularly in challenging classes. This work highlights the sensory gap in LLMs from a human-aligned perspective and proposes a principled approach to enhancing modality-specific perception in multimodal LLMs. 1 Introduction Humans perceive the world through multiple sensory systems, with the auditory and visual systems playing dominant roles. Each system is uniquely specialized: the auditory pathway is sensitive to temporal cues and fine-grained frequency patterns [ 1 ] , while the visual pathway is attuned to spatial structure and motion [ 2 ] . To the brain, audio stimuli travel from the cochlea to the auditory cortex, and visual stimuli travel from the retina to the visual cortex. These distinct routes reflect fundamental differences in how each modality encodes and interprets the environment. Yet, the two systems are interconnected at higher cortical levels, enabling multi-sensory integration for robust perception, such as recognizing a barking dog from its shape or sound. Recent advances in multimodal large language models (LLMs) [ 3 , 4 ] have introduced architectures that parallel human sensory systems. Audio and visual LLMs typically begin with a modality-specific encoder, such as Whisper [ 5 ] for audio or Vision Transformer (ViT) [ 6 ] for images, that transforms continuous sensory inputs into contextual representations. These representations, along with textual prompts, are then processed by a shared Transformer decoder-only language model [ 7 , 8 ] . This unified model operates analogously to the association cortices of the brain, where signals from different sensory pathways are integrated to support higher-level reasoning. Despite strong performance, the sensory boundaries and complementarity of audio and visual LLMs remain unexplored, especially in direct comparison to identical acoustic scenes. Studies in neuroscience and psychology show that human sensory systems exhibit modality-specific biases: audition excels in dynamic or occluded environments, while vision dominates in spatially rich scenes [ 9 , 10 ] . We ask whether a sensory gap exists in LLMs and, if so, whether it mirrors that observed in humans. Once we discover this modality-specific performance gap between audio and visual LLMs, another natural question arises: can we reduce the sensory gap by leveraging another modality? This question is inspired by the synesthesia phenomenon in humans [ 11 , 12 ] , where stimulation of one sensory can evoke perceptual experiences in another—suggesting the possibility of cross-sensory reinforcement. To answer both questions, this study is structured into two parts: analysis and distillation . In the analysis part, we evaluate audio LLM Qwen2-Audio [ 13 ] , visual LLM Qwen2-VL [ 14 ] , and audio-visual LLM Qwen2.5-Omni [ 15 ] , among other multimodal LLMs and real humans, in the audio-visual VGGSound dataset [ 16 ] . Each model receives audio-only, silent video, or both and is prompted to classify the sounding object from available sensors. This setup can reveal modality-specific strengths and weaknesses and eventually shows that a sensory gap exists for LLMs and aligns with this for humans, with some classes where Qwen2-Audio still underperforms human listeners. In the distillation part, we propose a cross-modal framework to distill knowledge between LLMs. A heuristic switch , trained on our analysis findings, routes supervision from the stronger to the weaker modality. Finally, our distilled Qwen2-Audio achieves over 20% accuracy gain, surpassing its visual teacher and matching the audio-visual Qwen2.5-Omni. In summary, our study highlights two key insights: first, audio and visual LLMs exhibit a sensory gap that mirrors that of humans, and second, these gaps can be effectively reduced through targeted cross-modal knowledge distillation. Together, these findings offer a deeper understanding of modality specialization in LLMs and a principled framework for enhancing their perceptual capabilities. Fig. 1: We compared Qwen2-Audio, Qwen2-VL, and Qwen2.5-Omni among themselves and against humans on 30 classes \times 50 samples/class from VGGSound analysis set under three input conditions of the same acoustic scene: audio-only, silent video, and sounded video. Class-wise analysis reveals that the sensory gap among LLMs (audio vs. visual vs. both) mirrors human perception differences between ears, eyes, and their integration. The overall accuracy of 30 classes shown in this figure is: Qwen2.5-Omni (94.4%) > Human Ears&Eyes (84.2%) > Qwen2-VL (83.9%) > Human Eyes (74.1%) > Qwen2-Audio (72.5%) > Human Ears (71.9%). Despite slightly higher overall accuracy, Qwen2-Audio underperforms human listeners in most challenging sound classes (bottom row). 2 Related Works (Visible) sound recognition has been studied across both audio and visual modalities. Models such as PANN [ 17 ] and AST [ 18 ] for audio, and ViT [ 6 ] and SAM [ 19 ] for images have set state-of-the-art results. More recent methods like CLIP [ 20 ] and CLAP [ 21 ] integrate language models for open-vocabulary recognition, producing free-form captions rather than selecting from a pre-defined label set. Audio and Visual LLMs represent the latest evolution in sound recognition, while their capabilities extend to broader scene analysis and reasoning tasks [ 3 , 4 ] . They typically integrate an audio or visual encoder, like those mentioned above, with a pretrained LLM. While prior studies have evaluated audio or visual LLMs individually [ 22 , 23 ] , and a few for audio-visual LLMs [ 24 , 25 ] , to our knowledge, this is the first work to directly contrast audio and visual LLMs and humans on the same acoustic scenes. Knowledge Distillation for LLMs typically happens between two text-only LLMs or two multimodal LLMs of the same modalities [ 26 , 27 ] . Only a few non-LLM works explore cross-modal distillation between audio and video models [ 28 , 29 ] . 3 Cross-Modal Analysis 3.1 Multimodal LLMs and Data This study focuses on single-sensory multimodal LLMs that perceive the environment with either the auditory or visual modality, but not both. To this end, our research subjects are three state-of-the-art audio LLMs: Qwen2-Audio [ 13 ] , Qwen-Audio [ 30 ] , and Kimi-Audio [ 31 ] , and three state-of-the-art visual LLMs: Qwen2-VL [ 14 ] , Qwen2.5-VL [ 32 ] , and VideoLLaMA3 [ 33 ] . Our analysis places particular emphasis on Qwen2-Audio and Qwen2-VL, as they both share Qwen2 [ 34 ] as the language model, allowing us to isolate and investigate

Read the complete text