Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models

Junyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jinming Liu, Xin Jin, Xiaoyu Shen · 2026

Read Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models by Junyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jinming Liu, Xin Jin, Xiaoyu Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06843.txt.

Introduction

Research by Junyan Lin and co-authors was posted on arXiv in 2026. It deals with speak while watching. The preprint joined the arXiv PAPERS collection in 2026. Multimodal Large Language Models (MLLMs) have achieved strong performance across many tasks. Yet most systems remain limited to offline inference, requiring complete inputs before generating outputs. Recent streaming methods reduce latency by interleaving perception and generation, but still enforce a sequential perception–generation cycle, limiting real-time interaction. For specialists, the preprint offers a compact account of the problem and the authors' approach.

Catalog record

CREATORJunyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jinming Liu, Xin Jin, Xiaoyu Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models Junyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jinming Liu, Xin Jin, Xiaoyu Shen Abstract Multimodal Large Language Models (MLLMs) have achieved strong performance across many tasks, yet most systems remain limited to offline inference, requiring complete inputs before generating outputs. Recent streaming methods reduce latency by interleaving perception and generation, but still enforce a sequential perception–generation cycle, limiting real-time interaction. In this work, we target a fundamental bottleneck that arises when extending MLLMs to real-time video understanding: the global positional continuity constraint imposed by standard positional encoding schemes. While natural in offline inference, this constraint tightly couples perception and generation, preventing effective input–output parallelism. To address this limitation, we propose a parallel streaming framework that relaxes positional continuity through three designs: Overlapped, Group-Decoupled, and Gap-Isolated. These designs enable simultaneous perception and generation, allowing the model to process incoming inputs while producing responses in real time. Extensive experiments reveal that Group-Decoupled achieves the best efficiency–performance balance, maintaining high fluency and accuracy while significantly reducing latency. We further show that the proposed framework yields up to 2× acceleration under balanced perception–generation workloads, establishing a principled pathway toward speak-while-watching real-time systems. We make all our code publicly available: https://github.com/EIT-NLP/Speak-While-Watching . 1 Introduction † † * Equal contribution. † Corresponding authors. Modern Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in a wide range of tasks 17 ; 1 ; 46 . However, the vast majority of existing systems still operate under an offline inference paradigm, in which the model must first ingest the entire input sequence before producing any output. While this design aligns well with current benchmark settings 12 ; 11 ; 25 , it inherently precludes real-time understanding and response. In practical, safety- and time-critical scenarios, such as assistive navigation 15 , sign language interpretation 6 , and live video description 8 , continuous perceptual feedback is essential. Systems that rely on offline processing cannot react promptly to dynamic changes in the environment, limiting their usability in real-world deployments. Figure 1 : Illustration of different paradigms for video description and positional encoding.The first row shows the offline paradigm, where the model generates the description after observing the entire video, leading to temporal misalignment between narration and visual sequence. The second row presents the interleaved streaming paradigm, which alternates between perception and generation, providing more immediate responses and better temporal coherence, but still suffers from the continuity constraint of positional encoding that prevents full parallelism. The third row illustrates our proposed parallel streaming paradigm, which breaks this continuity, enabling simultaneous perception and generation for true real-time video understanding. The positional IDs serves as a conceptual reference, showing how relaxing positional continuity enables parallel processing between input and output. Please zoom in for a clearer view of details. To address this, recent studies 8 ; 28 ; 43 have attempted to extend MLLMs into a streaming paradigm. However, most of these approaches are essentially interleaved: they alternately process a segment of input and then generate a segment of output. Although this reduces latency compared to fully offline inference, it still behaves like “mini-batch offline” processing and fails to achieve true real-time interaction . For example, in assistive navigation for the visually impaired, the system may be generating a long descriptive output about the next steps to take. If a sudden obstacle or danger appears during this period, an interleaved streaming MLLM, which performs perception and generation alternately rather than concurrently, may fail to detect the hazard in time. Such behavior is clearly unacceptable in safety-critical applications. These limitations stem from the fundamental constraints of decoder-only architectures 10 ; 40 ; 5 , which are not designed for simultaneous encoding and decoding. Although encoder–decoder architectures 29 ; 16 could, in principle, support such parallelism, converting existing MLLMs to this paradigm is highly impractical, as it would require re-establishing large-scale vision–language alignment from scratch. We argue that the true bottleneck lies not in the architecture itself, but in the positional encoding design . Current MLLMs enforce a global continuity constraint in positional indexing 32 ; 17 ; 37 . Because future output length is unknown at inference time, the model cannot assign consistent positional indices to incoming inputs while decoding is in progress, thereby preventing concurrent perception and generation. As illustrated in Fig. 1 , the first row depicts the offline setting for a video description task 4 , where the model generates the full description after observing the entire video. This often results in temporal misalignment between the narrative and the actual video sequence. For example, the model may start describing “shrimp cooking” even though the shrimp does not appear until the middle of the video. The second row illustrates the interleaved streaming setting, where the description follows the temporal order more naturally, first describing the melting of butter, then the cooking of shrimp, and finally the addition of seasonings. Although this result is more temporally coherent, its latency remains suboptimal because the continuity of positional encoding prevents the model from encoding the next incoming frame until the current text generation is completed. We observe that such strict positional continuity is not fundamentally required. The essential role of positional encoding is to capture relative relationships among tokens, rather than to impose a single, globally continuous index space 34 ; 33 . This insight allows us to decouple positional assignments across input and output streams while preserving the relational structure necessary for multimodal alignment. Motivated by this perspective, we introduce a parallel streaming paradigm that breaks positional continuity and enables true simultaneous encoding and decoding. As illustrated in the third row of Figure 1 , our approach allows the model to prefill embeddings for incoming visual frames during text generation, achieving real-time synchronization between perception and response. Specifically, we propose three intuitive positional encoding strategies— Overlapped Streaming Position Encoding (OSPE) , Group-Decoupled Position Encoding (GDPE) , and Gap-Isolated Position Encoding (GIPE) . In the OSPE strategy, the model begins encoding the next video frame concurrently with text decoding, assigning the same initial positional indices to both the current response and the next incoming frame. The GDPE strategy, in contrast, separates the input and output streams, assigning each its own positional group that starts from zero independently. Finally, the GIPE strategy extends the group-based design by adding a large numerical offset between input and output positions, thus creating an explicit separation in index space. We conduct extensive experiments under both offline and streaming inference paradigms on Video Description (VD) and Video Question Answering (VQA) tasks, systematically evaluating the proposed positional encoding strategies from the perspectives of performance, robustness, and acceleration potential. Empirically, we observe that in both offline and streaming settings, the original positional embedding scheme can be replaced by our proposed alternatives with only minimal fine-tuning data, while preserving comparable performance across standard evaluation metrics. In terms of robustness, we introduce scheduling perturbations at test time by disrupting the wait-K policy and find that all three proposed strategies consistently yield more fluent and stable language generation than conventional interleaved encoding. Considering both task accuracy and linguistic coherence, Group-Decoupled Position Encoding (GDPE) emerges as the most balanced and effective design.

Read the complete text