OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding

Keda Tao, Wenjie Du, Bohan Yu, Weiqiang Wang, Jian liu, Huan Wang · 2025

Read OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding by Keda Tao, Wenjie Du, Bohan Yu, Weiqiang Wang, Jian liu, Huan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23646.txt.

Introduction

Keda Tao and five co-authors built OmniAgent, a system that answers questions about videos by actively looking and listening rather than passively scanning frames. Models that read both audio and video often miss fine cross-modal detail and struggle to align the two streams. The arXiv record's title governs here; the catalog entry folds in the system's name differently.

OmniAgent plans its own investigation, calling specialized single-modality tools when needed and concentrating attention on task-relevant cues. Its audio-guided strategy works coarse to fine: sound first localizes interesting moments in time, then visual reasoning examines them closely. Without any training, the agent beat leading open and closed models on three audio-video benchmarks by margins reported between ten and twenty percentage points of accuracy.

Catalog record

CREATORKeda Tao, Wenjie Du, Bohan Yu, Weiqiang Wang, Jian liu, Huan Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding Keda Tao, Wenjie Du, Bohan Yu, Weiqiang Wang, Jian liu, Huan Wang Abstract Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often lack the fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, a fully audio-guided active perception agent that dynamically orchestrates specialized tools to achieve more fine-grained audio-visual reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, this paper demonstrates a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and proprietary models by substantial margins of 10% - 20% accuracy. https://kd-tao.github.io/OmniAgent Figure 1 : (a) : Illustration of O m n i A g e n t , an audio-guided active perception agent designed for omnimodal understanding. Given a complex user query, our agent employs a recursive “Think-Act-Observe-Reflect” loop and actively orchestrates multimodal tools (video, audio, and event tools) for fine-grained audio-video understanding. The presented video clip is from a real-life vlog; the question is about two Chinese characters on a hanging signboard in the video. Initially, the agent utilizes acoustic cues to locate the temporal segment containing the key information (“the kitten”), then invokes the video clip tool within that time window. Within the salient segment, given the same GPU memory budget, we can afford model inference at an increased spatial and temporal resolution. With sufficient relevant visual evidence and the audio as input, the agent derives the correct answer. In contrast, the end-to-end video understanding model Qwen3-Omni ( 49 ) (b) cannot achieve such a fine-grained understanding and gives the wrong answer. (c) : Performance comparison on three audio-video understanding benchmarks. OmniAgent demonstrates superior performance, consistently outperforming strong end-to-end OmniLLMs such as Qwen3-Omni ( 49 ) and Gemini 2.5-Flash ( 9 ) . 1 Introduction Recently, end-to-end omnimodal large language models (OmniLLMs) have achieved encouraging results by integrating visual and audio encoders into a unified architecture ( 37 ; 59 ; 48 ; 49 ; 51 ; 16 ; 34 ; 52 ; 34 ; 1 ) . Despite this progress, as shown in Figure 2 (a), OmniLLMs still struggle with in fine-grained cross-modal understanding, and the joint alignment training of audio and video representations poses significant challenges ( 14 ; 7 ; 36 ) . Consequently, models often cannot respond accurately. A critical empirical observation is that while MLLMs have demonstrated exceptional proficiency in unimodal tasks ( 42 ; 39 ; 3 ; 41 ; 8 ; 10 ; 2 ) , cross-modal understanding remains constrained by the challenges of temporal and feature alignment ( 7 ; 20 ; 11 ) . Consequently, developing agent-based frameworks to synergize the capabilities of distinct modalities is now a promising direction. Although preliminary investigations into agentic audio-visual understanding exist ( 4 ; 60 ) , current approaches rely predominantly on static workflows, as illustrated in Figure 2 (b). These methods not only incur substantial computational overhead but also face challenges in fully leverage the inherent thinking capabilities of models for dynamic planning, thereby precluding genuine fine-grained understanding. Recent works have yielded significant advancements in agent-based video understanding ( 58 ; 57 ; 12 ; 43 ; 53 ; 45 ; 32 ; 46 ; 6 ; 54 ) . Specifically, temporal event localization is paramount for fine-grained analysis. Prevailing approaches predominantly rely on frame-captioning, where captions are generated for sampled frames, stored, and subsequently retrieved and analyzed iteratively by agents ( 43 ; 58 ; 45 ) . While these methods refine their hypotheses through multi-step inference, they incur substantial computational overhead. Sometimes, even worse, the generated caption of the frames may be irrelevant to the query. However, in the context of audio-visual understanding, the audio modality presents distinct challenges yet offers a unique opportunity: unlike redundant visual signals, audio naturally provides accurate and concise temporal cues regarding the salient events ( 17 ; 38 ) . These cues can efficiently guide agent-based event localization , thereby facilitating a more accurate fine-grained understanding at low cost. Figure 2 : (a) End-to-end OmniLLMs implicitly fuse modalities but suffer from high training costs, difficult alignment, and limited fine-grained reasoning. (b) Fixed workflow agents rely on rigid pipelines, lacking the flexibility to allocate attention for fine-grained analysis adaptively. (c) Caption-based agents incur high precomputation costs and noise sensitivity, often failing to capture comprehensive multimodal context. (d) Our OmniAgent employs active perception reasoning and inquiry. Within an iterative reflective loop, the agent strategically calls on the ability of video and audio understanding. This explicitly solves the cross-modal alignment difficulty and achieves fine-grained understanding. Building on these insights, we present OmniAgent , an agent specifically designed for omnimodal (audio-visual) understanding in an autonomous and active reasoning fashion, which treats strong single-modal models as callable tools. In contrast to previous methods constrained by fixed workflows, as shown in Figure 2 (d), our approach initiates a fundamental paradigm shift from passive response generation to active information inquiry. The agent employs an LLM as the central component to orchestrate tool invocation, determining the optimal modality to use – it explicitly decides whether to attend to audio or video, and to what extent and how to understand. Note, this process is completely autonomous , decided by the LLM itself. The tool calling and decision making is transparent to us, thus explainable and optimizable. By strategically selecting spatial and temporal focus range ( i.e. , deciding precisely where to look and listen), OmniAgent achieves genuine, fine-grained cross-modal understanding. Specifically, we construct a comprehensive library of tools categorized into three distinct sets: (1) Video tools, (2) Audio tools, and (3) Event tools. The video toolset enables global captioning and general visual QA, while also allowing for the analysis of specific temporal windows at higher sampling rates to support more fine-grained understanding. The audio toolset incorporates audio captioning and detailed QA capabilities, complemented by timestamped automatic speech recognition (ASR) for precise speech grounding. Within the event toolset , we propose a novel audio-based event localization strategy. This mechanism empowers the agent to autonomously query and temporally localize events across the entire audio stream, establishing temporal anchors for subsequent fine-grained analysis. By synthesizing the capabilities of distinct MLLMs, our agent adaptively leverages their complementary strengths to facilitate joint, fine-grained analysis, utilizing cross-modal corroboration to maximize audio-visual comprehension performance. Extensive experiments show that OmniAgent achieves the best accuracy on all audio-video understanding benchmarks, surpassing the state-of-the-art open-source and closed-source models, such as Qwen3-Omni-30B ( 49 ) and Gemini2.5-Flash ( 9 ) , by a significant margin of 10-20% accuracy. The main contributions of this work are: • We introduce OmniAgent , a novel agent-based framework tailored for comprehensive audio-video understanding. Employing an active perception strategy, it dynamically modulates attention between auditory and visual modalities, and, via a self-reflective mechanism, solves the cross-modal alignment problem. •

Read the complete text