From Instruction to Event: Sound-Triggered Mobile Manipulation
Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng · 2026
Read From Instruction to Event: Sound-Triggered Mobile Manipulation by Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21667.txt.
Introduction
Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, and Zhedong Zheng let mobile robots act on sounds instead of waiting for commands. Most mobile manipulation research is instruction-driven: a person states a task and the robot obeys. That setup keeps the robot passive, reacting only when told.
Real homes and wards are full of events worth acting on, like a kettle whistling or a glass breaking. The authors trigger robot action from sound, so the robot notices and responds on its own. Event-driven action is a step toward helpers that share awareness of the room, not just obedience to text.
Catalog record
| CREATOR | Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21667
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Instruction to Event: Sound-Triggered Mobile Manipulation Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng Abstract Current mobile manipulation research predominantly follows an instruction-driven paradigm, where agents rely on predefined textual commands to execute tasks. However, this setting confines agents to a passive role, limiting their autonomy and ability to react to dynamic environmental events. To address these limitations, we introduce sound-triggered mobile manipulation, where agents must actively perceive and interact with sound-emitting objects without explicit action instructions. To support these tasks, we develop Habitat-Echo, a data platform that integrates acoustic rendering with physical interaction. We further propose a baseline comprising a high-level task planner and low-level policy models to complete these tasks. Extensive experiments show that the proposed baseline empowers agents to actively detect and respond to auditory events, eliminating the need for case-by-case instructions. Notably, in the challenging dual-source scenario, the agent successfully isolates the primary source from overlapping acoustic interference to execute the first interaction, and subsequently proceeds to manipulate the secondary object, verifying the robustness of the baseline. Figure 1 : Motivation for our work . (Upper) An example of sound-triggered mobile manipulation, the trigger signal is the sound of the event rather than instructions from humans. (Left) Instruction-based mobile manipulation needs humans to analyze the trigger signal and manually give the instruction for the robot model, which is passive. (Right) In sound-triggered mobile manipulation, the robot model automatically receives sound signals and actively outputs executable actions. 1 Introduction Recent advances in Embodied AI have witnessed remarkable progress in mobile manipulation, enabling robots to navigate unstructured environments and interact with physical objects ( 31 ; 3 ; 46 ) . Such synergy between mobility and interaction empowers agents to perform long-horizon household tasks such as object rearrangement ( 13 ; 11 ; 8 ) and multi-stage navigation ( 32 ; 22 ) , bringing us closer to general-purpose home robotic assistants. Current paradigm predominantly studies mobile manipulation under an instruction-conditioned setting, where the task goal and intended interaction are explicitly specified by a natural language instruction ( 40 ; 29 ; 42 ; 48 ) . While effective for user-specified chores, this paradigm is ill-suited for the stochastic nature of daily life. For example, events such as a boiling kettle or a ringing doorbell typically happen unexpectedly and require timely intervention, where the latency inherent in human-in-the-loop interaction renders timely intervention impossible. This uncertainty is further compounded when multiple events occur concurrently or in rapid succession, i.e . , waiting for a phone call from friends while expecting a doorstep delivery from a courier simultaneously. The indeterminate sequence of such events prevents the user from formulating a valid instruction plan in advance. This limitation underscores the discrepancy between static, predefined instructions and dynamic, sound-triggered scenarios (See Figure 1 ). Although recent efforts have attempted to generalize instructions ranging from concrete commands ( 44 ) to abstract goals ( 39 ) and ambiguous queries ( 12 ; 21 ) , addressing these stochastic events remains an open challenge. In this context, we find that sound signals play a pivotal role. For instance, if the phone rings prior to a knock at the door, the robot should prioritize picking up the phone before retrieving the delivery. In such cases, robotic actions are modulated by stochastic sounds rather than predefined text instructions. Specifically, sounds serve a threefold function: (1) temporal trigger indicating the start of a stochastic event; (2) semantic cue guiding the robot’s action selection; and (3) spatial cue revealing the event’s location. Based on these insights, we introduce sound-triggered mobile manipulation, comprising SonicStow, SonicInteract, and Bi-Sonic Manipulation, where acoustic signals replace conventional textual instructions as the trigger signal. SonicStow and SonicInteract present unique challenges beyond traditional visual manipulation: (1) Acoustic semantic understanding, which requires the robot to interpret sounds to determine target locations and actions. (2) Sound-conditioned interaction, which demands the execution of articulated interactions solely triggered by audio. The Bi-Sonic Manipulation task further elevates the complexity by introducing dual sound sources, which demands the agent to prioritize the primary source amidst acoustic interference and subsequently locate and interact with the secondary sound source. To validate our proposed tasks, we introduce Habitat-Echo , a simulation platform that seamlessly integrates acoustic rendering with interactive physical environments. Built upon the visual and physical capabilities of Habitat ( 34 ; 27 ) , Habitat-Echo addresses the absence of sound signals in prior simulators. Specifically, we leverage room impulse responses to convolve source sound with the environment’s geometry, thereby ensuring physically accurate sound propagation and spatialization from the source to the receiver. Leveraging this platform, we construct a benchmark tailored for sound-triggered mobile manipulation. The dataset encompasses many interactive assets, categorized into: (1) static silent rigid objects, (2) sound-emitting rigid objects, and (3) sound-emitting articulated objects. We utilize this dataset to train policy models and further introduce an Omni-modal Large Language Model (Omni-LLM) as a high-level task planner to understand acoustic events and decompose complex manipulation goals. The main contributions of this work are summarized as follows: • Novel Sound-Triggered Tasks. Distinct from traditional instruction-following paradigms, we formulate three novel manipulation tasks triggered by stochastic acoustic signals. In these tasks, robotic actions are autonomously triggered by stochastic sound events rather than static predefined instructions, better reflecting the unpredictable and dynamic nature of real-world household scenarios. • New Audio-Visual-Physical Simulation Platform. We introduce Habitat-Echo, a platform that integrates acoustic and visual rendering with interactive physical environments. By incorporating sound-emitting objects, Habitat-Echo facilitates large-scale episode generation and closed-loop online interaction, establishing a robust foundation for future research in embodied audio-visual learning. • Hierarchical Baseline and Validation. We develop a hierarchical framework that is composed of a high-level task planner to interpret acoustic cues and several low-level executors for precise control. We leverage an Omni-LLM as the task planner and policy models as executors. The baseline framework achieves promising performance on the proposed benchmark. Figure 2 : Illustration of the proposed tasks. (a) SonicStow requires the agent to interact with the sound source (a rigid object) via Navigate, Pick, and Place skills. (b) SonicInteract requires the agent to interact with the sound source (an articulated object) via Navigate, Open Door, and Close Sink skills. (c) Bi-Sonic Manipulation requires the agent to interact with the sound source (two objects sequentially) via Navigate, Pick, Place, Open Door, and Close Sink skills. 2 Related Work Sound Navigation. Sound Navigation, proposed by ( 6 ) , requires an agent to navigate to an audio source without explicit goal coordinates. The primary challenges involve understanding physical spaces, localizing sound emitters, and avoiding obstacles. Existing methods can be broadly categorized into two streams: real-world application and versatile training. For real-world application, researchers aim to bridge the simulation-to-reality gap by introducing complex environmental settings. ( 5 ) assign semantic meanings to audio sources and introduce dynamic sound emission, replacing the continuous, generic signals used in prior work. ( 45 ) further extends the setting from static to moving sound sources. To simulate complex acoustic environments, ( 47 ) introduces an adversarial sound attacker that generates distracting noises with varying categories and volumes. For versatile training, methods leverage cross-modal synergies to enhance performance. VXN ( 38