DiTalker: A Unified DiT-based Framework for High-Quality and Speaking Styles Controllable Portrait Animation
He Feng, Yongjia Ma, Donglin Di, Lei Fan, Tonghua Su, Member, IEEE , and Xiangqian Wu, Senior Member, IEEE · 2025
Read DiTalker: A Unified DiT-based Framework for High-Quality and Speaking Styles Controllable Portrait Animation by He Feng, Yongjia Ma, Donglin Di, Lei Fan, Tonghua Su, Member, IEEE , and Xiangqian Wu, Senior Member, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06511.txt.
Introduction
He Feng and co-authors build DiTalker, a tool that makes a still face talk in a 2025 paper. The aim is a clip that moves with a voice track, keeps the lips in step, and wears the talk style of a cue clip, from mood to head pose.
Past tools tend to nail the lips or freeze the mood, and they run on a pair of nets that costs time. DiTalker's one-net frame reads style and mood through two arms: one takes the talk style of the face at hand, the next takes mood in a face-free way. A fuse step keeps voice and style apart as it steers the frames. Two checks guard lip step and fine face marks. Tests put the tool ahead on both scores.
Catalog record
| CREATOR | He Feng, Yongjia Ma, Donglin Di, Lei Fan, Tonghua Su, Member, IEEE , and Xiangqian Wu, Senior Member, IEEE |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06511
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiTalker: A Unified DiT-based Framework for High-Quality and Speaking Styles Controllable Portrait Animation Thanks: * Corresponding authors: Tonghua Su ([email protected]), Lei Fan ([email protected]).He Feng, Tonghua Su, and Xiangqian Wu are with Harbin Institute of Technology, Harbin 150001, China (e-mail: [email protected]; [email protected]; [email protected]).Yongjia Ma and Donglin Di are with Li Auto, Beijing 101300, China (e-mail: [email protected]; [email protected]).Lei Fan is with University of New South Wales, Sydney 2052, Australia (e-mail: [email protected]). He Feng, Yongjia Ma, Donglin Di, Lei Fan, Tonghua Su, Member, IEEE , and Xiangqian Wu, Senior Member, IEEE Abstract Portrait animation aims to synthesize talking videos from a static reference face, conditioned on audio and style frame cues ( e.g., emotion and head poses), while ensuring precise lip synchronization and faithful reproduction of speaking styles. Existing diffusion-based portrait animation methods primarily focus on lip synchronization or static emotion transformation, often overlooking dynamic styles such as head movements. Moreover, most of these methods rely on a dual U-Net architecture, which preserves identity consistency but incurs additional computational overhead. To this end, we propose DiTalker, a unified DiT-based framework for speaking style controllable portrait animation. We design a Style-Emotion Encoding Module that employs two separate branches: a style branch extracting identity-specific style information like head poses and movements, and an emotion branch extracting identity-agnostic emotion features. We further introduce an Audio-Style Fusion Module that decouples audio and speaking styles via two parallel cross-attention layers, using these features to guide the animation process. To enhance the quality of results, we adopt and modify two optimization constraints: one to improve lip synchronization and the other to preserve fine-grained identity and background details. Extensive experiments demonstrate the superiority of DiTalker in terms of lip synchronization and speaking style controllability. Project Page: https://thenameishope.github.io/DiTalker/ . I Introduction Diffusion-based video generation has achieved remarkable progress in the inherently multimodal domains of Text-to-Video (T2V) and Image-to-Video (I2V) synthesis [ 1 , 2 , 3 ] . By demonstrating the capability to produce high-quality and temporally coherent videos, these methods have stimulated widespread exploration in diverse downstream applications [ 4 , 5 , 6 ] . Among them, human-centric video generation, notably portrait animation, has emerged as a prominent direction [ 7 , 8 , 9 ] due to its potential applications in video conferencing, film industry, and Virtual Reality. Portrait animation [ 10 ] aims to synthesize talking face videos from a static reference face, conditioned on driving signals such as audio and style frames, as shown in Fig. 1 . Fig. 1: Given a reference face, driving audio, and a specific speaking style, DiTalker can generate high-quality talking face videos. The waveform on the left represents the driving audio, with the images above each audio clip showing different speaking styles. In the generated videos on the right, we select three temporally discontinuous frames with significant variations to demonstrate the expressiveness and vividness of the results. In the real world, each individual has a distinct speaking style. Different people exhibit noticeable facial variations when speaking the same sentence, and the same individual may also show different facial dynamics when uttering it under different expressions. In the context of portrait animation, there are two key challenges: (1) the precise alignment of the generated video, which involves extracting lip movement from the driving audio [ 11 ] , and (2) the controllability of speaking style [ 12 ] , which involves transferring facial expressions, head poses, and head movements. Early studies [ 13 , 14 , 15 ] have made promising progress in lip synchronization, and recent works [ 8 , 16 ] have begun to explore speaking style controllable portrait animation. For instance, InstructAvatar [ 16 ] uses emotion prompts to guide emotional expression, and MoEE [ 17 ] employs emotion-specific experts for different emotions to achieve fine-grained emotional control. SLIGO [ 9 ] uses audio features to control both head pose and emotional expressions. HEAD [ 7 ] utilizes a prediction network to estimate 3DMM parameters [ 18 ] and drive facial deformation for style control. These approaches can be categorized into three groups based on their methodology and resulting limitations: (a) those [ 17 , 16 ] solely guided by emotion templates or experts, which fail to capture identity-specific speaking styles; (b) those [ 19 , 20 , 9 ] imprecisely predicting head pose and expressions from audio features; and (c) those [ 21 , 22 , 7 ] relying on statistical parametric models like 3DMM, thereby limiting generalization and emotional expressiveness. As illustrated in Fig. 2 , the comparison of inference results under the same source face, driving audio, and speaking style reveals the inherent limitations of previous methods. These limitations include facial distortions, a lack of expressive emotions, and monotonous head poses and movements, and they compromise the naturalness and vividness of the generated talking videos. Fig. 2: Comparison of our DiTalker with three categories of portrait animation methods: (a) EDTalk [ 23 ] , (b) Echomimic [ 19 ] , and (c) SAAS [ 24 ] . DiTalker demonstrates superior controllability of speaking style. Recent diffusion-based portrait animation methods such as EMO [ 25 ] and Echomimic [ 19 ] employ a dual U-Net architecture, including a Reference Net that is used to extract fine-grained features of the reference source face, and a Denoising Net that is responsible for generating results. This architectural design achieves more natural and high-quality head movements in generated talking face videos, significantly outperforming previous GAN-based approaches [ 7 , 13 , 26 , 27 ] . While this design effectively ensures identity consistency, the introduction of a Reference Net incurs additional computational and training overhead. Moreover, current research lacks investigation into applying single-diffusion architectures, particularly Diffusion Transformer (DiT) [ 28 ] , which has shown superior performance over U-Net counterparts in both T2V and I2V [ 29 ] generation tasks. This gap highlights the opportunity to explore whether a single DiT-based architecture can not only reduce architectural complexity and training cost but also maintain, or even enhance, generation quality while keeping identity consistency for portrait animation. In light of the above limitations, we propose DiTalker , a unified DiT-based framework for speaking style controllable portrait animation, conditioned on a static reference face, driving audio, and optional style frames. DiTalker comprises three key components: a DiT generation backbone, a Style-Emotion Encoding Module (SEEM), and an Audio-Style Fusion Module (ASFM). Specifically, SEEM is designed to explicitly extract speaking style features through two parallel branches. The style branch processes style frames and phonemes extracted from driving audio to generate style embeddings, while the emotion branch encodes T5-extracted [ 30 ] emotional prompts from the same frames to produce emotion embeddings. Notably, unlike prior methods like EAT [ 31 ] , TalkCLIP [ 32 ] (employing a separate emotion branch), and StyleTalk++ [ 12 ] (employing a separate style branch), our SEEM comprises emotion and style branches. This design explicitly disentangles the head pose and movements derived from style frames from the expressed emotions, enabling independent control over each. The style branch models identity-specific speaking styles, including eye, mouth, and head movements, from style frames. Concurrently, the emotion branch models identity-agnostic emotions via an emotion template design. This architecture ensures both the preservation of