FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait

Taekyung Ki, Dongchan Min, Gyeongsu Chae · 2024

Read FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait by Taekyung Ki, Dongchan Min, Gyeongsu Chae (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01064.txt.

Description

Taekyung Ki, Dongchan Min, and Gyeongsu Chae present FLOAT, a system that animates portraits from speech, in this two thousand twenty-four paper. Diffusion-based portrait animation produces strong results, but its iterative sampling makes videos temporally inconsistent and slow to generate. FLOAT instead builds on flow matching, a generative approach that samples more directly. It also works in a learned orthogonal motion latent space rather than a pixel latent space, which supports efficient generation and editing of consistent motion.

A transformer-based vector field predictor, conditioned frame by frame, drives the motion synthesis. The method additionally supports speech-driven emotion enhancement, folding expressive motion into the performance naturally. Extensive experiments showed FLOAT beating state-of-the-art audio-driven talking portrait methods on visual quality, motion fidelity, and efficiency. The work targets applications from virtual presenters to film dubbing, where a still portrait must speak convincingly.

Catalog record

CREATORTaekyung Ki, Dongchan Min, Gyeongsu Chae
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01064

Browse the full library · About the Tower of Babel