Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation
Shihao Cheng ⋆, Jiaxu Zhang ⋆, Quanyue Song, Shansong Liu †, Zhizhi Guo, Xiaolei Zhang, Chi Zhang, Xuelong Li ‡, Zhigang Tu ‡ · 2026
Read Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation by Shihao Cheng ⋆, Jiaxu Zhang ⋆, Quanyue Song, Shansong Liu †, Zhizhi Guo, Xiaolei Zhang, Chi Zhang, Xuelong Li ‡, Zhigang Tu ‡ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08729.txt.
Introduction
Shihao Cheng and eight coauthors present Unison, a generator for human centered video where motion, speech, and sound effects must line up. These streams run on unlike time patterns, so earlier models often mismatch lips, movement, and background sound. Unison splits speech from sound effects inside the audio stream, guiding their reunion with semantic cues so speech no longer drowns out other sounds.
For picture and sound sync, the cleaner stream guides the noisier one through staggered denoising steps, steadied progressively. Experiments place Unison at the top for audio quality and cross stream alignment. Video generation gains a lesson: coherence must be engineered explicitly, not hoped for.
Catalog record
| CREATOR | Shihao Cheng ⋆, Jiaxu Zhang ⋆, Quanyue Song, Shansong Liu †, Zhizhi Guo, Xiaolei Zhang, Chi Zhang, Xuelong Li ‡, Zhigang Tu ‡ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation Shihao Cheng ⋆, Jiaxu Zhang ⋆, Quanyue Song, Shansong Liu †, Zhizhi Guo, Xiaolei Zhang, Chi Zhang, Xuelong Li ‡, Zhigang Tu ‡ Abstract Motion, speech, and sound effects are fundamental elements of human - centric videos, yet their heterogeneous temporal characteristics make joint generation highly challenging. Existing audio - video generation models often fail to maintain consistent alignment across these modalities, leading to noticeable mismatches between motion, speech, and environmental sounds. We present Unison, a unified framework that explicitly promotes coherence across the motion, speech, and sound modalities. Within the audio stream, Unison employs a semantic-guided harmonization strategy that decouples the generation of speech and sound-effect components. Leveraging bidirectional audio cross-attention and semantic-conditioned gating for semantic-driven adaptive recomposition, this approach effectively mitigates speech dominance and enhances acoustic clarity. For audio–motion synchronization, we propose a bidirectional cross-modal forcing strategy where the cleaner modality guides the noisier one through decoupled denoising schedules, reinforced by a progressive stabilization strategy. Extensive experiments demonstrate that Unison achieves state - of - the - art performance in both audio perceptual quality and cross - modal synchronization, highlighting the importance of explicit multimodal harmonization in human - centric video generation. \NoHyper † † footnotetext: ⋆ Equal contribution. † Project leader. ‡ Corresponding author. \endNoHyper Figure 1 : Overview of the key challenges and our approach. Left: Two major misalignments in human-centric audio-video generation—(a) speech-sound interference within the audio stream and (b) motion-audio desynchronization, including motion-speech and motion-sound mismatches. Middle: Our proposed Unison framework addresses these issues through a semantic-guided harmonization strategy that decouples and adaptively balances speech-sound components via interactive refinement, and a progressive audio-video forcing strategy that enforces rigorous temporal alignment across modalities. Right: These designs jointly reduce speech dominance, preserve sound effects, and achieve coherent synchronization between motion and audio. 1 Introduction The rapid evolution of generative AI [ 2 , 45 , 36 , 22 , 17 , 10 , 18 , 29 , 16 , 46 , 47 ] has catalyzed significant breakthroughs in synchronized audio-video generation. Proprietary models such as Veo 3 [ 9 ] , Sora 2 [ 24 ] , and Wan 2.5 [ 36 ] , alongside open-source models like Ovi [ 22 ] , UniAVGen [ 44 ] , and LTX-2 [ 10 ] , have demonstrated remarkable creative potential, inspiring users across online communities. Although most open-source approaches have continuously improved visual fidelity [ 37 , 14 ] , they still exhibit poor alignment between audio and video streams. Recent methods, such as Harmony [ 11 ] , employ fine-grained cross-attention mechanisms to enable frame-level synchronization. However, these approaches primarily focus on architectural modifications and lack explicit consistency constraints. In contrast, closed-source commercial systems leverage large-scale datasets to achieve fully data-driven synchronization [ 24 ] , facilitating visually and temporally coherent outputs. Nevertheless, the generated audio often lacks acoustic richness and balance: speech components tend to dominate the entire mix, suppressing ambient and contextual sounds that are essential for perceptual realism. This imbalance leads to flattened auditory scenes and weakens the overall sense of immersion, with these issues becoming especially evident in human-centric video generation. As shown in the left part of Figure 1 , our evaluation of representative audio–visual generation models reveals two fundamental challenges. (1) Intra - audio interference: speech and non - speech sound effects often overlap within the same audio stream, creating perceptual masking where structured speech components dominate and obscure transient environmental sounds—especially in human - centric scenarios such as “singing while playing an instrument." This indicates that current models lack mechanisms to disentangle and balance heterogeneous acoustic elements. (2) Cross - modal misalignment: motion and audio generation are typically linked only through architectural cross - attention, without explicit synchronization objectives during training. Consequently, audio and video exchange information implicitly but fail to form consistent temporal correspondence, leading to desynchronized gestures, lip motions, and acoustic onsets. These issues collectively limit the realism and coherence of human - centered video generation. To overcome the aforementioned challenges, we propose Unison, a unified framework designed to generate motion, speech, and sound in a coherent and synchronized manner. To resolve acoustic interference, Unison implements a semantic-guided harmonization strategy that decouples speech and sound-effect generation. This approach integrates bidirectional audio cross-attention (Bi-ACA) for interactive refinement and semantic-conditioned gating (SCG) for semantic-driven adaptive recomposition. By modulating component interplay, Unison ensures cross-modal consistency while enabling dedicated modeling for each acoustic entity, effectively preventing speech from overshadowing subtle environmental sounds. Second, to tackle motion–audio desynchronization, Unison employs a bidirectional cross-modal forcing paradigm that explicitly aligns denoising progress between modalities. By allowing the temporally advanced modality to guide the lagging one during training, the model learns stable and causally consistent motion–audio coordination without relying solely on cross-attention. Together, these designs establish a principled way of generating human-centric videos where motion, speech, and acoustic context evolve in Unison–yielding perceptually balanced and temporally synchronized results. Extensive experiments verify that Unison not only enhances overall generation fidelity but also achieves markedly better harmony among motion, speech, and sound. The resulting videos exhibit more balanced auditory layers and tighter temporal correspondence between modalities, confirming that both spectral disentanglement and cross - modal alignment substantially improve perceptual realism. Moreover, the framework’s symmetric design naturally enables bidirectional generation—it can synthesize video conditioned on audio or, conversely, generate audio from visual cues—demonstrating its flexibility as a unified model for multimodal content creation. The main contributions are summarized as follows: • We propose Unison, a unified audio–video generation framework that produces motion, speech, and sound in a coherent and harmonized manner. Unlike prior models that loosely couple audio and visual streams, Unison explicitly enforces cross - modal consistency, achieving balanced and temporally aligned outputs. • We propose a semantic-guided harmonization strategy that decouples speech and sound-effect generation to resolve acoustic interference. By integrating Bidirectional Audio Cross-Attention (Bi-ACA) for interactive refinement and Semantic-Conditioned Gating (SCG) for semantic-driven adaptive recomposition, resulting in controllable and acoustically layered audio generation. • We introduce a bidirectional cross-modal forcing strategy that strengthens temporal dependencies between audio and motion. By allowing the modality in a clearer denoising state to guide the other, Unison achieves robust synchronization and stable training, effectively bridging the gap between motion and acoustics in human - centric video generation. 2 Related Work 2.1 Joint Audio-Video Generation Early joint audio-video generation, exemplified by MM-Diffusion [ 27 ] , utilized U-Net architectures for ambient sound synthesis, a scope later expanded by large-scale dataset training [ 19 , 49 ] . While UniVerse-1 [ 37 ] and Ovi [ 22 ] recently introduced human speech generation, their reliance on implicit global alignment often compromises temporal precision. Advanced mechanisms [ 11 , 44 , 10 ] have since achieved frame-level synchronization via fine-grained cross-attention. Nonetheless, these methods typically employ unidirectional conditioning, leaving bidirectional cross-modal reinforcement and the acoustic interference between speech and effects largely unaddressed. Recent commercial systems, including Veo 3 [ 9 ] , Sora 2 [ 24 ] , and Wan 2.5 [ 36 ] , have achieved cinematic audio-video synchronization. Nevertheless, when synthesizing scenes with concurrent dense speech and sound effects, these models often prioritize human voices, leading to the suppression or