PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers
Xiangyue Zhang Yiyi Cai Kunhang Li Kaixing Yang You Zhou, Zhengqing Li Xuangeng Chu Jiaxu Zhang Haiyang Liu · 2026
Read PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers by Xiangyue Zhang Yiyi Cai Kunhang Li Kaixing Yang You Zhou, Zhengqing Li Xuangeng Chu Jiaxu Zhang Haiyang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06064.txt.
Introduction
Xiangyue Zhang and eight co-authors personalize co-speech gestures for new speakers from a single reference clip. An avatar should move with the new speaker's habits, yet one clip mixes stable personal style with motion tied to that one utterance. No per speaker retraining is allowed in their setting, which makes the split harder.
PersonaGesture attacks the split with two parts working in sequence. A style perceiver compresses the reference into speaker memory tokens that guide the diffusion based generator as it forms motion. A second step then corrects leftover channel statistics in latent space, scaled to the reference length. On two gesture data sets, this separation beat collapsed style codes, full reference attention, and one clip fine tuning.
Catalog record
| CREATOR | Xiangyue Zhang Yiyi Cai Kunhang Li Kaixing Yang You Zhou, Zhengqing Li Xuangeng Chu Jiaxu Zhang Haiyang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06064
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers Xiangyue Zhang Yiyi Cai Kunhang Li Kaixing Yang You Zhou, Zhengqing Li Xuangeng Chu Jiaxu Zhang Haiyang Liu Abstract We propose PersonaGesture, a diffusion-based pipeline for single-reference co-speech gesture personalization of unseen speakers. Given target speech and one motion clip from a new speaker, the model must synthesize gestures that follow the new utterance while retaining speaker-specific pose choices, without per-speaker optimization. This setting is useful for avatars and virtual agents, but it is hard because the reference mixes stable speaker habits with utterance-specific trajectories. PersonaGesture consists of two key components, Adaptive Style Infusion (ASI) and Implicit Distribution Rectification (IDR), to separate temporal identity evidence from residual statistic correction. A Style Perceiver first encodes the variable-length reference into compact speaker-memory tokens. ASI injects these tokens into denoising through zero-initialized residual cross-attention, enabling style evidence to affect motion formation without replacing the pretrained speech-to-motion prior. Building on this, IDR applies a length-aware diagonal affine map in latent space to correct residual channel-wise moments estimated from the same reference. Across BEAT2 and ZeroEGGS, we evaluate quantitative metrics, reference-identity controls, same-audio diagnostics, qualitative comparisons, and human preference. Experiments show that separating denoising-time speaker memory from conservative post-generation moment correction improves unseen-speaker personalization over collapsed style codes, full-reference attention, and one-clip finetuning. Project: https://xiangyue-zhang.github.io/PersonaGesture . 1 Introduction Co-speech gestures make avatars, digital actors, and virtual agents feel tied to a person rather than to a generic motion prior [ 9 , 45 , 49 , 48 ] . Recent speech-driven gesture models can produce plausible motion for speakers that appear in their training set [ 54 , 39 , 43 , 38 , 81 , 68 ] . Deployment is different. A new user may only provide a short recording before the system has to speak with that user’s gesture style. Collecting a full motion-capture set is costly, and per-user optimization adds delay and can overfit the recording. We study this practical personalization problem from one reference clip. The speaker is unseen during training, the target speech is different from the reference utterance, and the model does not update its parameters at test time. Figure 1 : Motivation and deployment contrast. (a) Optimization-based personalization requires person-specific data collection and a per-person adaptation loop. (b) PersonaGesture instead receives one reference clip and new target speech at test time, caches a compact style memory and reference moments, and uses them through ASI and IDR. (c) The cached reference evidence controls generation for new speech, producing identity-consistent gesture motion without test-time parameter updates. Recent diffusion-based co-speech gesture generators have improved motion realism, rhythm, and audio alignment on fixed speaker sets [ 83 , 5 , 67 , 27 , 21 , 40 , 44 ] . However, these models are not directly equipped for single-reference personalization. Many systems learn speaker IDs from training identities, use a compact style code, or adapt the model on a new speaker at test time. These designs become brittle when the only evidence is one held-out clip. The clip contains useful cues about gesture timing, amplitude, spatial preference, and holds. It also contains motion that is specific to the recorded sentence. A personalizer must therefore read the reference as identity evidence. It should not use the reference as a trajectory template for the new speech. Figure 1 shows this deployment contrast and previews the two reference signals used by our method. The first difficulty is how to use temporal evidence. Speaker style is not fully captured by an average pose vector. Two speakers may have similar marginal pose statistics but differ in gesture onset, hold length, amplitude growth, and preferred hand space. Mean pooling tends to remove this information. Full-sequence reference attention has the opposite problem. It gives the denoiser direct access to utterance-specific trajectories, which can encourage copying. A useful reference path needs a structured bottleneck. It should keep recurring temporal cues while hiding the full trajectory from the generator. The second difficulty is how far to correct the generated distribution. One clip gives noisy estimates of speaker-level statistics. It is too small for stable per-speaker finetuning when the target utterances differ from the reference. It is also too small for high-capacity distribution matching. Full-covariance or higher-order corrections can fit the observed clip too closely. Ignoring the statistics also leaves a mismatch between generated latents and the target speaker. The correction should therefore be conservative. It should use low-order moments that one clip can estimate, and it should shrink when the reference is short. To address these challenges, we propose PersonaGesture, a diffusion-based pipeline for single-reference gesture personalization. PersonaGesture uses the reference in two roles. The first role is style control during denoising. A Style Perceiver maps the reference latents into a fixed number of speaker-memory tokens. Same-speaker contrastive supervision makes this memory identity-aware. The second role is residual correction after generation. The same reference latents provide channel-wise means and variances for a conservative affine map. This separation is the main design choice. Temporal speaker evidence enters while gestures are being formed. Low-order statistics are corrected only after the sequence has been generated. Adaptive Style Infusion (ASI) implements the temporal pathway. ASI inserts the speaker memory into the frozen diffusion backbone through zero-initialized residual cross-attention. The denoising hidden states query the compact memory at each layer, allowing the reference to bias timing, amplitude, and motion-space decisions without replacing the speech-to-motion prior. Zero initialization makes the augmented denoiser equal to the pretrained backbone at the start of Stage 2, so training learns a residual speaker-conditioning path rather than relearning gesture generation. This design contrasts with full-reference attention, which exposes all reference tokens. It also contrasts with mean-pooled conditioning, which removes temporal structure needed for speaker-specific dynamics. Implicit Distribution Rectification (IDR) implements the residual-statistics pathway. After ASI-conditioned generation, IDR applies a diagonal affine map in latent space using the generated sequence moments and the reference moments. The map is motivated by a limited diagonal-Gaussian analysis of residual channel-wise mismatch. This analysis does not claim that the full gesture distribution is Gaussian, nor does it prove optimality of the generator. It only supports the rectifier used after denoising. Because reference statistics are finite-sample estimates, IDR uses a length-aware shrinkage rule. Short references receive weaker correction, while longer references allow a stronger moment match. Section 4.1 gives the exact evaluation setting. The main question is not whether a larger generator can make better gestures in general. It is whether the single reference adds target-speaker information that survives new speech. Our experiments therefore compare matched reference paths, one-clip adaptation baselines, reference-identity controls, split and seed checks, reference-length stress tests, qualitative examples, and human preference. The results show that the compact denoising memory and the conservative moment rectifier are complementary. Our contributions can be summarized as: • We study no-update co-speech gesture personalization from a single reference clip, where the speaker is unseen and the target speech is different from the reference utterance. • We propose a two-part reference pathway. ASI injects a compact speaker memory during denoising, and IDR applies length-aware latent moment correction after generation. • Experiments on BEAT2 and ZeroEGGS show stronger unseen-speaker personalization than collapsed style codes, full-reference attention, and one-clip finetuning baselines. 2 Related