ARIG: Autoregressive Interactive Head Generation for Real-time Conversations

Ying Guo, Xi Liu, Cheng Zhen, Pengfei Yan, Xiaoming Wei Vision AI Department, Meituan https://jinyugy21.github.io/ARIG/ · 2025

Read ARIG: Autoregressive Interactive Head Generation for Real-time Conversations by Ying Guo, Xi Liu, Cheng Zhen, Pengfei Yan, Xiaoming Wei Vision AI Department, Meituan https://jinyugy21.github.io/ARIG/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00472.txt.

Introduction

Ying Guo and co-authors generate a talking head that chats in real time in a paper from two thousand twenty-five. A virtual partner should listen and speak with apt face motion. Older methods worked clip by clip or swapped between a listener and a speaker model, causing lag and jumps.

Their system, ARIG, predicts motion frame by frame. A diffusion step models motion in smooth space instead of picking from a codebook. One part reads short-range cues from both partners' sound and motion, and another tracks talk states like pauses, feedback, and interruptions. These feed the final motion guess. Tests verified the model's gains in realism and speed.

Catalog record

CREATORYing Guo, Xi Liu, Cheng Zhen, Pengfei Yan, Xiaoming Wei Vision AI Department, Meituan https://jinyugy21.github.io/ARIG/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00472

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ARIG: Autoregressive Interactive Head Generation for Real-time Conversations Ying Guo, Xi Liu, Cheng Zhen, Pengfei Yan, Xiaoming Wei Vision AI Department, Meituan https://jinyugy21.github.io/ARIG/ Abstract Face-to-face communication, as a common human activity, motivates the research on interactive head generation. A virtual agent can generate motion responses with both listening and speaking capabilities based on the audio or motion signals of the other user and itself. However, previous clip-wise generation paradigm or explicit listener/speaker generator-switching methods have limitations in future signal acquisition, contextual behavioral understanding, and switching smoothness, making it challenging to be real-time and realistic. In this paper, we propose an autoregressive (AR) based frame-wise framework called ARIG to realize the real-time generation with better interaction realism. To achieve real-time generation, we model motion prediction as a non-vector-quantized AR process. Unlike discrete codebook-index prediction, we represent motion distribution using diffusion procedure, achieving more accurate predictions in continuous space. To improve interaction realism, we emphasize interactive behavior understanding (IBU) and detailed conversational state understanding (CSU). In IBU, based on dual-track dual-modal signals, we summarize short-range behaviors through bidirectional-integrated learning and perform contextual understanding over long ranges. In CSU, we use voice activity signals and context features of IBU to understand the various states (interruption, feedback, pause, etc.) that exist in actual conversations. These serve as conditions for the final progressive motion prediction. Extensive experiments have verified the effectiveness of our model. 1 Introduction Face-to-face communication is a common human activity around us [ 32 ] . Research on head generation makes it possible for virtual agents to become communication participants. Early works focus on single-role generation (talking/listening). In talking head generation, the input signal comes only from the speaker himself and does not receive signals from the other participant for interaction. The listening head generation receives signals from both participants to provide appropriate listener responses, but it only offers non-verbal motion feedback and cannot generate speaker responses. In real conversations, each participant’s response should integrate the behaviors of both parties and possess the ability to both speak and listen. Thus, single-role generation cannot fully simulate behaviors in conversations. Figure 1 : Development of interactive head motion generation. a) Only speaker generation based on last speaking behaviors. b) Different explicit generators for listener/speaker. c) Implicit clip-by-clip generator based on a unified model. d) Implicit frame-by-frame generator based on a unified non-quantized AR model. Recently, some works [ 27 , 39 , 30 , 37 , 29 , 19 ] have studied the interactive head generation (IHG), as shown in Fig. 1 . Suppose the conversation between a user and a virtual agent, IHG generates interactive head responses for the agent by utilizing available information from both parties. Initially, early works ( Fig. 1 a) such as MRecGen [ 30 ] and [ 19 ] model the conversation as a multi-turn mode and only generate the agent’s speaker response based on the user’s speaking behavior in the last turn. Then, methods ( Fig. 1 b) in [ 37 ] , DIM [ 27 ] and AgentAvatar [ 29 ] can generate interactive responses of both speakers and listeners. However, they have to assign explicit roles (speaker/listener) for the agent and switch between two respective generators. Conversations involve more complex states beyond speaker/listener alternation ( e.g . interruption, pause, giving verbal feedback), requiring immediate and smooth role switching, and sometimes even both parties play the same role, which is difficult to achieve with explicit role switching. To solve this, in Fig. 1 c, INFP [ 39 ] uses a unified generator to switch roles implicitly and dynamically. Although its inference speed is ideal, it is a clip-wise paradigm that inputs two fixed-length clips ( T\!\!\sim\!\!T\!+n ) of audio and output motions within T\!\!\sim\!\!T\!+n . At time T , since the signal about the future T\!+\!n cannot be acquired, the model must wait until the audio is acquired at T\!+\!n and then forms a clip to generate the motion at T , which will cause a delay of at least one clip, making real-time generation difficult. Besides, for interaction realism , the clip-wise paradigm limits the visible range to short clips. Without long context, it may misinterpret the logic of statements and the person’s intent, resulting in unreasonable motions. Moreover, its input in motion guider contains only audio without visual information, which may not be sufficient to fully represent the behavior of a conversation participant, who may interact in many non-audio ways such as frowning and nodding. To solve these problems, we propose a novel framework called ARIG based on A uto R egression to redefine the I nteraction G eneration paradigm which is real-time and has better interaction realism . For real-time generation , we design a frame-wise generation framework ( Fig. 1 d) which generates the T - th frame promptly in an autoregressive (AR) manner based on previous T - 1 frames, which can respond in time, without future clip being available. We implement it with a continuous non-quantized AR, enjoying the advantages of sequential processing and more accurate motion representation. For better interaction realism , we emphasize the impact of contextual semantics by combining long- and short-range features in AR sequences and integrate dual-track (user and agent) dual-modal (audio and visual motion) signals to understand interactive behaviors. Furthermore, we analyze complex conversion states (interrupted, feedback, wait, overlap, pause, etc.), noting that even with the same audio, the motions differ in different states. For example, for “wow it’s amazing to…”, in a normal speaking state, one might widen eyes to express exclamation, while in a state of giving feedback to the other party, nodding may be performed as approval. Thus, we learn the state features as a guide for motion generation. Specifically, to achieve real-time frame-wise generation with higher fidelity , we model the framework as non-quantized autoregression (AR) for continuous-valued predictions. Unlike conventional AR motion modeling that predicts discrete and finite codebook indices, we use a diffusion procedure to model the probability distribution of motion, enabling the prediction of continuous values to reduce deviation of discretization and better reproduce micro-expressions. To better understand the contextual interaction , we design an interactive behavior understanding module (IBU). Considering the learning difficulties and time costs of excessive long-range information, we first effectively summarizes dual-modal dual-track interaction by bidirectional-integrated learning within a short chunk, and perform contextual understanding over a long range. To comprehensively learn complex conversation states , we design a conversation states understanding module (CSU), which combines the voice activity signal and the contextual feature of IBU module to learn current conversation states. Finally, we conduct a progressive motion prediction (PMP) conditioned on context, conversation states, and audio, combining past motions to generate final motions. In summary, this paper has the following contributions: • We propose a novel frame-wise generation framework based on continuous AR with diffusion probability modeling, which formulates the real-time paradigm of interactive head generation with higher motion fidelity. • We design an IBU module to learn the contextual interaction, which effectively summarize dual-track dual-modal signals of short range via bidirectional-integrated learning, and capture contextual information over a long range. • We analyze the complex states in real-world conversations in more detail, and design a CSU module to learn the state

Read the complete text