Hierarchical Codec Diffusion for Video-to-Speech Generation
Jiaxin Ye, Gaoxiang Cong, Chenhui Wang, Xin-Cheng Wen, Zhaoyang Li, Boyuan Cao, Hongming Shan · 2026
Read Hierarchical Codec Diffusion for Video-to-Speech Generation by Jiaxin Ye, Gaoxiang Cong, Chenhui Wang, Xin-Cheng Wen, Zhaoyang Li, Boyuan Cao, Hongming Shan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15923.txt.
Introduction
Ye and six colleagues propose HiCoDiT, a hierarchical codec diffusion transformer that creates speech from silent video. Speech has layers, from coarse speaker-aware meaning to fine prosodic detail, and older methods ignore this hierarchy. HiCoDiT uses low-level blocks keyed to lip motion and facial identity for speaker content, and high-level blocks keyed to facial expression for prosody. A dual-scale normalization layer blends global vocal style with local prosody dynamics, and tests show it beats baselines on fidelity and expressiveness.
Catalog record
| CREATOR | Jiaxin Ye, Gaoxiang Cong, Chenhui Wang, Xin-Cheng Wen, Zhaoyang Li, Boyuan Cao, Hongming Shan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41195 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15923
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Codec Diffusion for Video-to-Speech Generation Jiaxin Ye, Gaoxiang Cong, Chenhui Wang, Xin-Cheng Wen, Zhaoyang Li, Boyuan Cao, Hongming Shan Abstract Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT , a novel Hi erarchical Co dec Di ffusion T ransformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT . 1 Introduction Video-to-Speech (VTS) Ye et al. 2025 ; Cong et al. 2023 ; Cong et al. 2025a generation aims to infer and synthesize speech from visual cues alone. This capability enables transformative applications, including silent film dubbing and assistive communication for aphonic individuals, to seamless interaction in noise-sensitive Darur and Singla 2025 , privacy-critical Potamianos et al. 2004 or embodied Feng et al. 2026 environments. The fundamental challenge in VTS lies in addressing the inherent information asymmetry between visual and acoustic modalities when generating natural and lip-synchronized speech from visual without acoustic input guidance. Specifically, although facial video and speech share consistent content Radford et al. 2023 , identity Ravanelli et al. 2021 , and emotional prosody Ye et al. 2023b ; Ye et al. 2023a ; Wen et al. 2022 , visual features are inherently sparse and insufficient to capture the dense representations of speech, making it difficult to build accurate cross-modal alignment. Existing approaches predominantly focus on representation alignment for guiding generative models Choi et al. 2023 ; Goto et al. 2020 ; Jang et al. 2024 ; Chen et al. 2022 , spanning semantic content, vocal identity, and emotional prosody from vision to speech: ( i ) for semantic content alignment, NaturalL2S Liang et al. 2026 leverages multimodal self-supervised representations to enhance the alignment between visual semantics and speech content; ( ii ) for vocal identity alignment, Face2Speech Goto et al. 2020 aligns features from a face recognition encoder and a speaker recognition encoder to map facial identity to timbre information, while Face-StyleSpeech Kang et al. 2023 further incorporates contrastive learning to improve face-to-speech alignment; ( iii ) for emotional prosody alignment, FTV Kim et al. 2025 aligns facial emotion embeddings with pitch and energy to enhance prosody expressiveness. However, existing VTS methods typically inject visual features into holistic speech representations while overlooking the hierarchical structure of speech, from coarse speaker-aware semantics to fine-grained prosodic details, which ultimately exacerbates the inherent information asymmetry between visual and acoustic modalities. Therefore, the principal bottleneck in high-quality video-to-speech generation lies in visual conditioning, and how to exploit the speech hierarchy as a prior to improve generation quality remains unresolved. In this paper, we propose HiCoDiT , a novel Hi erarchical Co dec Di ffusion T ransformer that fully leverages the inherent hierarchy of discrete speech tokens to enable more effective vision-speech alignment. To the best of our knowledge, HiCoDiT is the first to introduce an explicit speech hierarchy prior into a discrete diffusion framework for video-to-speech generation. Specifically, leveraging the hierarchical structure of the Residual Vector Quantization (RVQ) codec in Figure 2 , the low-level tokens primarily capture rich speaker-aware semantic content, whereas the high-level tokens encode more abstract prosodic details. Therefore, the hierarchy prior dictates that visual features such as lip motion and facial identity should primarily refine low-level speech tokens, while facial emotion features should modulate high-level tokens. Motivated by this prior, we design a hierarchical codec diffusion transformer composed of low-level and high-level blocks, progressively conditioning on speech tokens across different levels. The low-level blocks generate tokens conditioned on synchronized lip-motion representations and facial identity features for semantic and timbre alignment, while the high-level blocks produce tokens guided by facial emotion sequences for prosody alignment. To achieve more effective conditioning in the high-level block, we introduce a dual-scale Adaptive Instance Layer Normalization (AdaLN) that employs channel-wise normalization to model global vocal style, and temporal-wise normalization to capture local prosody dynamics. Extensive experiments demonstrate that HiCoDiT surpasses state-of-the-art baselines in semantic alignment and expressive prosody. Our contributions are summarized as follows. • To our knowledge, HiCoDiT is the first discrete diffusion framework for VTS to explicitly integrate speech hierarchy prior, bridging the gap between video and speech. • We propose a novel hierarchical diffusion transformer that models the speech hierarchy while disentangling visual conditioning, and a dual-scale AdaLN to inject global vocal style and local prosody into speech generation, enhancing expressiveness and fidelity. • Extensive experiments demonstrate superior performance in semantic consistency and speech diversity, highlighting the potential of discrete speech tokens modelling for efficient VTS generation. 2 Related Work Video-to-Speech (VTS) generation. Video-to-speech (VTS) seeks to generate speech that accurately reflects both linguistic content and speaker identity from visual cues alone Choi et al. 2023 ; Yemini et al. 2024 ; Kim et al. 2023 . Current approaches typically enforce alignment through auxiliary objectives: some predict text or mel-spectrograms jointly with visual input Kim et al. 2023 , others condition speaker embeddings on lip motion Choi et al. 2023 or minimize cross-modal embedding distances Goto et al. 2020 . While effective in isolation, these methods treat speech as a flat sequence without hierarchy and impose multiple supervision signals, leading to suboptimal alignment. Recent advances have explored more sophisticated generative frameworks. For example, FTV Kim et al. 2025 employs flow matching with a hierarchical visual encoder to gradually inject visual features into continuous mel-spectrogram space, while VoiceCraft-Dub Sung-Bin et al. 2025 adapts pretrained autoregressive discrete text-to-speech models Peng et al. 2024 to incorporate visual context. However, both obscure the inherent hierarchical structure of speech representation, in which coarse linguistic content emerges at early token levels and fine prosodic detail is resolved later. In contrast, HiCoDiT introduces the first discrete diffusion model for VTS trained from scratch, which explicitly integrates the speech hierarchy prior, bridging the gap between video and speech. Hierarchical speech generation. Given the intrinsic hierarchy of speech, extensive research has focused on hierarchical representation modelling to achieve high-quality speech generation. In text-to-speech (TTS), Lee et al . Lee et al. 2022 propose a hierarchical conditional variational autoencoder (VAE) that leverages self-supervised speech representations to bridge the information gap between text and speech, and Hsu et al . Hsu et al. 2019 likewise propose a conditional VAE with two levels of hierarchical latent variables, which captures coarse acoustic information and refines specific