OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

Yaofeng Su ∗, Yuming Li ∗, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan · 2026

Read OmniForcing: Unleashing Real-time Joint Audio-Visual Generation by Yaofeng Su ∗, Yuming Li ∗, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11647.txt.

Introduction

Su and co-authors present OmniForcing, the first framework to turn offline audio-visual diffusion models into streaming generators. Bidirectional attention gives fine quality yet blocks real-time use. An asymmetric block-causal design with a prefix stops the two streams from drifting apart. An audio sink token tames gradient blow-ups from sparse audio tokens. Self-forcing distillation lets the model fix cross-modal errors over long rollouts, at about 25 frames per second on one GPU.

Catalog record

CREATORYaofeng Su ∗, Yuming Li ∗, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38504
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OmniForcing: Unleashing Real-time Joint Audio-Visual Generation Yaofeng Su ∗, Yuming Li ∗, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan Abstract Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at ∼ \sim 25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher. Project Page: https://omniforcing.com † † footnotetext: * Equal contribution. 1 Introduction The landscape of generative AI has been significantly advanced by Diffusion Transformers (DiTs) [ 30 , 40 ] . Building on this foundation, joint audio-visual models such as LTX-2 [ 7 ] and Veo 3 [ 44 ] have recently achieved notable progress, leveraging modality-specific VAEs [ 19 , 32 , 22 , 7 , 41 ] to map video and audio into continuous latent spaces and jointly model their temporal distribution. However, this capability comes at a substantial computational cost. These models rely heavily on bidirectional full-sequence attention, meaning the entire physical timeline must be processed simultaneously. Consequently, they suffer from a high Time-To-First-Chunk (TTFC) latency [ 51 ] , limiting their deployment in interactive, real-time, or streaming applications (see Fig. 1 for a comparison). Figure 1 : OmniForcing breaks the latency barrier for joint audio-visual generation. Top: Our framework achieves real-time streaming at \sim 25 FPS with an ultra-low Time-To-First-Chunk (TTFC) of \sim 0.7s. Bottom: The bidirectional teacher (LTX-2) requires \sim 197s to generate the sequence offline. OmniForcing maintains visual and acoustic fidelity on par with the teacher model. To mitigate this latency bottleneck, previous efforts have diverged into two primary workarounds. The first line employs cascaded pipelines, generating video first and subsequently synthesizing audio [ 28 , 2 , 43 , 29 ] (or vice versa) [ 37 , 13 ] . However, this decoupled paradigm severs the joint distribution, limiting generation quality and fundamentally obstructing continuous streaming since the secondary audio modality cannot begin until the primary video has materialized sufficient context. Another line of work adapts video-only diffusion models into causal, autoregressive frameworks (e.g., CausVid [ 51 ] , Self-Forcing [ 9 ] ), but these methods remain confined to the visual domain. Directly extending them to dual-stream architectures is non-trivial, as the severe temporal asymmetry between modalities induces a critical information deficit for the sparser modality [ 14 ] , leading to training instability rather than seamless stable alignment. To address these challenges, we present OmniForcing , the first framework to successfully distill a heavy, bidirectional audio-visual foundation model ( i.e . , LTX-2 [ 7 ] ) into a high-fidelity streaming autoregressive generator. By dynamically interleaving the generation of audio and video chunks, OmniForcing enables ultra-low latency streaming without sacrificing the holistic multi-modal distribution learned by the bidirectional teacher. Our framework tackles the modality asymmetry through a carefully designed Asymmetric Block-Causal Alignment , and the full three-stage distillation pipeline is illustrated in Fig. 2 . Firstly, to establish a persistent cross-modal anchor at the temporal origin, we introduce a zero-truncation Global Prefix that aligns the joint sequence at exact one-second boundaries (3 video frames to 25 audio frames) and exploits the native stride characteristics of the VAEs [ 7 , 19 ] . Then, to stabilize the perilous cross-modal causal shift, we propose an unsupervised Audio Attention Sink mechanism, inspired by the attention sink phenomenon in language [ 46 ] and vision [ 3 ] models. Based on this design, we create a position-agnostic global memory buffer by assigning an Identity RoPE [ 36 ] constraint to these sink tokens, which mitigates the gradient explosions inherent in sparse causal audio attention. Furthermore, to combat the exposure bias amplified by cross-modal error accumulation during long rollouts, we employ a joint Self-Forcing [ 9 ] distillation strategy that enables the model to dynamically self-correct. Finally, we exploit the intra-layer decoupling between the 14B video and 5B audio streams [ 7 ] by introducing a Modality-Independent Rolling KV-Cache that reduces per-step context complexity to \mathcal{O}(L) and enables concurrent execution of the two modality streams on a single GPU. In summary, our main contributions are: • We propose OmniForcing , a unified autoregressive framework that transforms offline, bidirectional joint audio-visual models into real-time streaming engines while preserving exact multi-modal temporal synchronization. • We introduce a natural Asymmetric Block-Causal Alignment and the Audio Sink Token mechanism with Identity RoPE, providing a robust, position-agnostic solution to the Softmax collapse caused by multi-modal token density mismatch. • We introduce a Modality-Independent Rolling KV-Cache with asymmetric parallel inference and a Joint Self-Forcing Distillation paradigm, which together mitigate exposure bias and reduce per-step context complexity to \mathcal{O}(L) , achieving state-of-the-art streaming generation at \sim 25 FPS on a single GPU. 2 Related Work Joint Audio-Visual and Video Foundation Models. The landscape of generative AI has been significantly advanced by large-scale Diffusion Transformers (DiTs) [ 30 , 40 ] . For visual generation, foundation models such as Sora [ 26 ] , Wan 2.1 [ 41 ] , HunyuanVideo [ 45 ] and Kling [ 38 ] have demonstrated high visual fidelity, physical realism, and adherence to complex text prompts. Building upon these unimodal visual successes, the field has recently achieved a notable shift toward unified multimodal generation. Joint audio-visual foundation models like LTX-2 [ 7 ] and Veo 3 [ 44 ] have emerged as state-of-the-art systems capable of generating highly synchronized, high-fidelity audio and video in a single pass. Notably, LTX-2 employs an asymmetric dual-stream architecture (a 14B video stream and a 5B audio stream) coupled through bidirectional cross-attention to deeply model the joint distribution of both modalities. While these foundation models deliver remarkable semantic alignment and generation quality, they exhibit a critical limitation regarding deployment: they rely exclusively on bidirectional full-sequence attention . Because the generation of a single frame requires the model to simultaneously attend to the entire physical timeline, the computational complexity scales quadratically with sequence length. This results in a massive Time-To-First-Chunk (TTFC) latency, rendering these models ill-suited for supporting real-time, interactive, or streaming applications. Audio-Visual Synthesis and Alignment. Prior to the emergence of joint foundation models, multi-modal generation heavily relied on cascaded or decoupled pipelines. These methods typically generate video first and subsequently synthesize the matching audio track using Foley sound generators ( e.g . , FoleyGen [ 29 ] , Diff-Foley [ 28 ] , FoleyCrafter [ 52 ] ,

Read the complete text