FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait
Taekyung Ki, Dongchan Min, Gyeongsu Chae · 2024
Read FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait by Taekyung Ki, Dongchan Min, Gyeongsu Chae (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01064.txt.
Description
Taekyung Ki, Dongchan Min, and Gyeongsu Chae present FLOAT, a system that animates portraits from speech, in this two thousand twenty-four paper. Diffusion-based portrait animation produces strong results, but its iterative sampling makes videos temporally inconsistent and slow to generate. FLOAT instead builds on flow matching, a generative approach that samples more directly. It also works in a learned orthogonal motion latent space rather than a pixel latent space, which supports efficient generation and editing of consistent motion.
A transformer-based vector field predictor, conditioned frame by frame, drives the motion synthesis. The method additionally supports speech-driven emotion enhancement, folding expressive motion into the performance naturally. Extensive experiments showed FLOAT beating state-of-the-art audio-driven talking portrait methods on visual quality, motion fidelity, and efficiency. The work targets applications from virtual presenters to film dubbing, where a still portrait must speak convincingly.
Catalog record
| CREATOR | Taekyung Ki, Dongchan Min, Gyeongsu Chae |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01064