ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
Xueyun Tian, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen · 2026
Read ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding by Xueyun Tian, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10323.txt.
Introduction
Xueyun Tian and co authors build a live aid that hears, sees, and reads at once. Such aids are named omni multimodal. They pledge one model for sound, sight, and text. But live streams are hard. Aids tend to lack full mode reach or fail to watch on their own.
The team made ROMA, a live aid for streams. It can react to what a user says. It can also act first when a stream shows a key event. That dual skill is named proactive watch. The aim is an aid that joins a live stream as a true peer.
Catalog record
| CREATOR | Xueyun Tian, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10323
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding Xueyun Tian, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen Abstract Recent Omni-multimodal Large Language Models show promise in unified audio, vision, and text modeling. However, streaming audio-video understanding remains challenging, as existing approaches suffer from disjointed capabilities: they typically exhibit incomplete modality support or lack autonomous proactive monitoring. To address this, we present ROMA , a real-time omni-multimodal assistant for unified reactive and proactive interaction. ROMA processes continuous inputs as synchronized multimodal units , aligning dense audio with discrete video frames to handle granularity mismatches. For online decision-making, we introduce a lightweight speak head that decouples response initiation from generation to ensure precise triggering without task conflict. We train ROMA with a curated streaming dataset and a two-stage curriculum that progressively optimizes for streaming format adaptation and proactive responsiveness. To standardize the fragmented evaluation landscape, we reorganize diverse benchmarks into a unified suite covering both proactive (alert, narration) and reactive (QA) settings. Extensive experiments across 12 benchmarks demonstrate ROMA achieves state-of-the-art performance on proactive tasks while competitive in reactive settings, validating its robustness in unified real-time omni-multimodal understanding. Our project page is available at here 1 1 1 https://eureka-maggie.github.io/ROMA_show/ . Figure 1: ROMA’s streaming understanding capabilities. It supports proactive tasks, including event alerts and narration, alongside reactive question answering. 1 Introduction Recent advances in omni-multimodal large language models (OLLMs), such as GPT-4o 16 , have enabled unified modeling of speech, vision, and text. This progress facilitates real-world streaming audio-video understanding, defined as combining reactive and proactive capabilities (Figure 1 ). In the reactive setting, the model answers after the query, whereas in the proactive setting, it follows an instruction to continuously monitor the input stream and respond only when conditions are met. Unifying these capabilities is vital for real-world utility, yet the divergent interaction paradigms make it challenging 14 ; 35 ; 10 . Despite the critical need for such unification, existing studies typically lack unified modality support and streaming capabilities. Specifically, speech-centric streaming models 8 ; 49 focus on audio generation but lack visual perception. Conversely, while some approaches address streaming video understanding 4 ; 48 , they typically neglect synchronized audio and are confined to specific tasks (e.g., alert or narration). Consequently, unified streaming audio-video understanding remains largely under-explored. To realize such unification faces two challenges. First, audio and video exhibit mismatched temporal granularities. While naturally synchronized, audio signals are dense and continuous, whereas video comprises sparse, discrete frames. Under such heterogeneity, maintaining robust cross-modal alignment and fusion demands precise synchronization. Second, effective streaming interaction requires real-time proactive decision-making. Upon integrating these asynchronous signals, the model must continuously synthesize context to determine both response timing and content, conditioned strictly on the stream prefix. To address these challenges, we propose ROMA , a R eal-time O mni- M ultimodal A ssistant with interactive streaming understanding. To tackle the granularity mismatch, ROMA segments continuous audio into one-second intervals synchronized with video frames, forming temporally aligned units that are processed sequentially as the stream unfolds. We further adapt chunked Time-aligned Multimodal RoPE (TMRoPE) 37 to enforce a shared temporal timeline. For proactive decision-making, ROMA introduces a lightweight speak head parallel to the standard language modeling (LM) head to explicitly predict response timing, decoupling timing from content generation to prevent task interference. Finally, we support this system with a custom streaming dataset and a two-stage training curriculum, progressively optimizing the model for cross-modal streaming format adaptation and proactive responsiveness. For a comprehensive evaluation, streaming audio-video understanding demands assessing both reactive and proactive capabilities. However, as compared in Table 1 , existing benchmarks suffer from inconsistent taxonomies and fragmented protocols, often failing to cover both interaction modes. To enable unified comparison, we reorganize the evaluation landscape into two standardized settings: a proactive mode that tests the ability to autonomously trigger responses at precise moments, and a reactive mode that emphasizes understanding temporal evolution in standard QA. Empirically, ROMA consistently outperforms existing streaming VideoLLMs across both modes. Furthermore, evaluations on open-ended audio-query QA against open-source OLLMs confirm its superior capability in unified audio-video understanding. In summary, our contributions are as follows: • Unified streaming framework: We formally define the task of streaming audio-video understanding and propose ROMA , an omni-multimodal assistant unifying reactive and proactive capabilities, supported by a curated dataset and a two-stage curriculum. • Standardized evaluation benchmark: We establish a comprehensive streaming benchmark by reorganizing fragmented tasks into unified reactive and proactive settings to facilitate rigorous and consistent comparison. • Superior performance and analysis: ROMA achieves state-of-the-art results across proactive benchmarks while competitive on reactive and open-ended QA. Extensive analysis verifies the efficacy of our timing mechanisms and training strategies. Benchmark Alert Narration Reactive QA StreamingBench 19 ✓ ✗ ✓ StreamBench 33 ✗ ✗ ✓ OVO-Bench 21 ✗ ✓ ✓ SVBench 42 ✗ ✗ ✓ OmniMMI 31 ✓ ✗ ✓ OVBench 15 ✗ ✗ ✓ Ours ✓ ✓ ✓ Table 1: Coverage of key streaming ability across representative streaming video benchmarks. Figure 2: Model Architecture. Streaming inputs are processed as aligned multimodal units. The speak head determines response timing, activating the LM head (illustrated via narration) upon crossing a probability threshold. 2 Related Works Reactive Models Most existing streaming systems are studied in the reactive setting, answering only after the query arrives. Within this regime, memory-based methods maintain long-range context for coherent understanding over evolving streams 23 ; 47 ; 32 ; 48 ; 36 ; 28 ; 50 , and KV-cache based methods optimize efficiency via scheduling or compression 9 ; 20 ; 40 ; 39 ; 7 . Recent omni-multimodal models also adhere to this reactive protocol: MiniCPM-o 2.6 46 , Qwen2.5-Omni 37 , and Qwen3-Omni 38 support low-latency interaction, and Stream-Omni 49 enables visually-conditioned speech generation, yet none explicitly model proactive monitoring and triggering. Proactive Models In contrast, proactive streaming prioritizes continuous monitoring and time-sensitive triggering (e.g., alerts and real-time narration). Proactive VideoLLMs leverage online formats or explicit decision modeling to determine intervention timing 4 ; 44 ; 18 ; 43 ; 22 , with some explicitly targeting live narration 5 . However, these approaches remain predominantly video-centric, neglecting streaming audio. Streaming Video Understanding Benchmarks Recent benchmarks prioritize time-sensitive, interactive evaluation (Table 1 ). StreamingBench 19 and OVO-Bench 21 assess temporal perception, while StreamBench 33 and SVBench 42 focus on long-horizon memory. Moreover, OmniMMI 31 and OVBench 15 incorporate proactive capabilities, including real-time narration and alerts. Tables 1 and 10 summarizes prior works. 3 Method To unify reactive answering and proactive timing over continuous inputs, ROMA integrates architectural designs with a tailored training strategy. Section 3.1 presents the model architecture, utilizing chunked TMRoPE and a speak head for alignment and timing control. Section 3.2 details the training and inference pipeline, encompassing dataset construction and a two-stage fine-tuning recipe. 3.1 Model Architecture As illustrated in Figure 2 , ROMA processes streaming omni-modal inputs via a unified LLM backbone. We introduce a speak head parallel to the LM head to decouple interaction timing from content generation. This architecture addresses temporal alignment and proactive decision-making through the following mechanisms. Multimodal units for temporally aligned streaming inputs. To support unified streaming understanding across modalities, we organize audio and video into fixed-interval multimodal