ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search

Zhenjie Liu, Jianzhang Lu, Renjie Lu, Cong Liang, Shangfei Wang · 2025

Read ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search by Zhenjie Liu, Jianzhang Lu, Renjie Lu, Cong Liang, Shangfei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06833.txt.

Introduction

Zhenjie Liu and four co-authors present ConsistTalk, a system that animates a still portrait so it appears to speak given audio. Video diffusion models have improved such talking-head creation, but results still flicker, drift away from the person's identity, and fall out of sync with the voice. ConsistTalk attacks all three problems together.

An optical-flow module separates motion from static appearance to cut flicker. A distilled audio-to-intensity model converts sound and facial speed into a frame-by-frame intensity sequence, giving fine control over how strongly the face moves while staying matched. Finally, a noise search at creation time enforces background coherence and motion continuity. Tests show clearer identity preservation and steadier, higher-quality video than earlier methods.

Catalog record

CREATORZhenjie Liu, Jianzhang Lu, Renjie Lu, Cong Liang, Shangfei Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search Zhenjie Liu, Jianzhang Lu, Renjie Lu, Cong Liang, Shangfei Wang Abstract Recent advancements in video diffusion models have significantly enhanced audio-driven portrait animation. However, current methods still suffer from flickering, identity drift, and poor audio-visual synchronization. These issues primarily stem from entangled appearance-motion representations and unstable inference strategies. In this paper, we introduce ConsistTalk , a novel intensity-controllable and temporally consistent talking head generation framework with diffusion noise search inference. First, we propose an optical flow-guided temporal module (OFT) that decouples motion features from static appearance by leveraging facial optical flow, thereby reducing visual flicker and improving temporal consistency. Second, we present an Audio-to-Intensity (A2I) model obtained through multimodal teacher-student knowledge distillation. By transforming audio and facial velocity features into a frame-wise intensity sequence, the A2I model enables joint modeling of audio and visual motion, resulting in more natural dynamics. This further enables fine-grained, frame-wise control of motion dynamics while maintaining tight audio-visual synchronization. Third, we introduce a diffusion noise initialization strategy (IC-Init) . By enforcing explicit constraints on background coherence and motion continuity during inference-time noise search, we achieve better identity preservation and refine motion dynamics compared to the current autoregressive strategy. Extensive experiments demonstrate that ConsistTalk significantly outperforms prior methods in reducing flicker, preserving identity, and delivering temporally stable, high-fidelity talking head videos. Introduction Audio-driven portrait animation aims to synthesize expressive talking head videos from a static portrait image and accompanying speech audio. Early works ( 40 ; 32 ) primarily employ a two-stage pipeline using 3D motion or keypoints as intermediate representations. In the first stage, audio features are extracted and mapped to motion latents, which are then used to generate video frames through a rendering network. However, representational bottlenecks limit the ability of these methods to produce dynamic and realistic facial movements. Recent advancements in this field have been spurred by the emergence of image generation diffusion models such as Stable Diffusion (SD) ( 23 ) and DiT ( 18 ) . Diffusion models have demonstrated superior generative capabilities compared to non-diffusion models, leading to their adaptation for portrait animation tasks. For example, EMO ( 28 ) , the first end-to-end framework capable of generating high-realism animations from a single image and audio, extends SD to video generation by incorporating temporal modules. Likewise, other notable approaches ( 2 ; 34 ; 11 ; 10 ; 31 ) built upon the pretrained image diffusion model backbone demonstrate robust generation capabilities in this domain. Figure 1: Failure cases in maintaining visual and temporal consistency of current methods. The left samples exhibit significant visual distortion (a) and background deformation with identity drift (b). The rightmost visualization displays temporal flickering (c) in the optical flows. Despite these advancements, existing methods still exhibit significant temporal and visual inconsistencies, as illustrated in Figure 1 . First, predominant approaches leverage motion features extracted from preceding frames through the ReferenceNet as temporal signals. However, flickering remains problematic due to the contamination of appearance information in complex scenes ( 5 ) . Second, current methods either predict motion solely from audio or from previous frames, resulting in abrupt temporal jitter. Controlling motion intensity may disrupt the synchronization between audio and visual elements, leading to unstable inferences. Finally, many methods ( 34 ; 5 ; 31 ) adopt an autoregressive-like inference strategy for long video generation to maintain consistent motion across segments. By recursively using generated frames as reference images, deformations will accumulate over time and cause noticeable appearance drift. To address these challenges, we propose ConsistTalk , a novel intensity-controllable and temporally consistent talking head generation framework with a diffusion noise search inference strategy. The method comprises three key innovations: 1) Optical Flow-guided Temporal Module (OFT) : We utilize facial optical flow to decouple facial dynamics from static appearance features, thereby preventing appearance contamination in the temporal layer and reducing flicker. 2) Audio-to-Intensity (A2I) Module : Intensity modeling captures the one-to-many mapping between audio and facial movements. A student model is distilled from a multimodal teacher to predict a frame-wise intensity sequence from audio alone, enabling controllable motion while maintaining audio-visual synchronization. 3) IC-Init strategy : We introduce a training-free inference algorithm that leverages a noise searching mechanism guided by the intensity sequence. By enforcing constraints on background stability and motion continuity, we enhance generation robustness and dynamic quality. Extensive experiments demonstrate that ConsistTalk outperforms existing methods in temporal consistency, identity preservation, and motion fidelity. Our contributions are summarized as follows. • We present ConsistTalk , a novel diffusion-based model that leverages facial optical flow to decouple motion from appearance, thereby mitigating temporal flickering. Through multimodal learning, it effectively captures the correlation between audio and motion velocity and enables motion dynamics control. • We propose IC-Init , a training-free diffusion noise initialization strategy that leverages an intensity-guided noise search, significantly improving identity preservation and motion continuity during inference. • We conduct fair and extensive experiments across diverse settings to validate our approach. Results show that ConsistTalk consistently generates more vivid, expressive, and temporally coherent talking head videos than existing state-of-the-art methods. Figure 2: Overall architecture of the proposed ConsistTalk framework. The system integrates three key modules: (a) a facial optical flow-guided temporal module (OFT) that decouples motion dynamics from appearance features to suppress flicker and enhance temporal consistency; (b) a audio-to-intensity (A2I) model that transforms audio and facial velocity features into frame-wise intensity signals for fine-grained motion control; and (c) a noise initialization strategy (IC-Init) that stabilizes the generation process by leveraging a noise search procedure. Related Work Audio-driven talking head generation has seen significant progress in recent years. Existing approaches generally fall into two categories based on their generation strategies: two-stage pipelines and end-to-end generation frameworks. Two-Stage Talking Head Generation. Early works like SadTalker ( 40 ) and Aniprotrait ( 32 ) in this field primarily incorporated intermediate 3D motion representations and employed a two-stage pipeline for video generation. Specifically, in the first stage, audio features are converted into motion parameters, while in the second stage, these motion cues guide a rendering or warping-based synthesis module to generate video frames. However, the reliance on intermediate motion representations limits their ability to model nuanced facial dynamics. End-To-End Talking Head Generation. Recent studies ( 28 ; 34 ; 5 ; 11 ; 10 ; 38 ) have integrated diffusion models into this field. By leveraging the powerful generative capabilities of pretrained image diffusion models, these approaches employ Stable Diffusion ( 23 ) or Wan 2.1 ( 30 ) as the backbone network. To enhance the image generation model’s ability to capture cross-frame dynamics, these methods incorporate ReferenceNet features from previous frames to construct temporal layers. However, because they rely on overall visual features as motion control signals without decoupling motion features, they still suffer from temporal inconsistencies. Figure 1 (c) illustrates cross-frame optical flow, showing sudden variations in the background that result in noticeable temporal flickering. In contrast, our method explicitly separates motion dynamics from appearance via facial optical flow, improving background coherence. This fundamentally differs from implicit decoupling strategies such as Patch-Drop in Hallo2 ( 5 ) . Secondly, current methods often struggle to achieve visually consistent control of motion intensity. Hallo (

Read the complete text